Generation and implementation of household policies for the smart home

ABSTRACT

Embodiments provided herein relate to implementing a household policy within a household environment. In one example, a method includes: receiving, at a processor, the household policy; interpreting the household policy to extract one or more conditional events associated with the household policy; monitoring, via at least one sensing smart device in the household environment, for satisfaction of the one or more conditional events; and when the one or more events is satisfied, implement one or more controls on at least one conditionally controlled smart device in the household environment, the at least one smart device affecting the household environment.

BACKGROUND

This disclosure relates to smart-home environments. In particular, thisdisclosure relates to generation and/or distribution ofdevice-implementable occupant policies for smart-device environments.

This section is intended to introduce the reader to various aspects ofart that may be related to various aspects of the present disclosure,which are described and/or claimed below. This discussion is believed tobe helpful in providing the reader with background information tofacilitate a better understanding of the various aspects of the presentdisclosure. Accordingly, it should be understood that these statementsare to be read in this light, and not as admissions of prior art.

People interact with a number of different electronic devices on a dailybasis. In a home setting, for example, a person may interact with smartthermostats, lighting systems, alarm systems, entertainment systems, anda variety of other electronic devices. Unfortunately, the usefulness ofthese devices often times limited to basic and/or particularpre-determined tasks associated with the device.

As society advances, households within the society may becomeincreasingly diverse, having varied household norms, procedures, andrules. Unfortunately, because so-called smart devices have traditionallybeen designed with pre-determined tasks and/or functionalities,comparatively fewer advances have been made regarding using thesedevices in diverse or evolving households or in the context of diverseor evolving household norms, procedures, and rules.

SUMMARY

A summary of certain embodiments disclosed herein is set forth below. Itshould be understood that these aspects are presented merely to providethe reader with a brief summary of these certain embodiments and thatthese aspects are not intended to limit the scope of this disclosure.Indeed, this disclosure may encompass a variety of aspects that may notbe set forth below.

According to embodiments of this disclosure, a smart-home environmentmay be provided with smart-device environment policies that usesmart-devices to monitor activities within a smart-device environment,report on these activities, and/or provide smart-device control basedupon these activities.

Various refinements of the features noted above may exist in relation tovarious aspects of the present disclosure. Further features may also beincorporated in these various aspects as well. These refinements andadditional features may exist individually or in any combination. Forinstance, various features discussed below in relation to one or more ofthe illustrated embodiments may be incorporated into any of theabove-described aspects of the present disclosure alone or in anycombination. The brief summary presented above is intended only tofamiliarize the reader with certain aspects and contexts of embodimentsof the present disclosure without limitation to the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of this disclosure may be better understood upon readingthe following detailed description and upon reference to the drawings inwhich:

FIG. 1 is a block diagram of a smart home device, in accordance with anembodiment;

FIG. 2 illustrates a connected smart home or smart device environmentthat includes a number of smart home devices, in accordance with one ormore embodiments;

FIG. 3 illustrates a network-level view of an extensible devices andservices platform with which the smart-home environment of FIG. 2 can beintegrated, according to an embodiment;

FIG. 4 illustrates an abstracted functional view of the extensibledevices and services platform of FIG. 3, with reference to a processingengine as well as devices of the smart-home environment, according to anembodiment;

FIG. 5 is a schematic drawing of a plurality of smart-deviceenvironments, in accordance with an embodiment;

FIG. 6 is a schematic drawing of a smart-home having a household policymanager that does not communicate over an external network, inaccordance with an embodiment;

FIG. 7 is a flow diagram illustrating a process for implementingpolicies based upon capabilities of smart-devices within the household,in accordance with an embodiment;

FIG. 8 is a schematic drawing of a household system capable ofimplementing the process of FIG. 7, in accordance with an embodiment;

FIG. 9 is a flow diagram illustrating a process for obtaining policiesbased upon demographic information, in accordance with an embodiment;

FIG. 10 is a schematic drawing illustrating a system that infersdemographic information to obtain relevant household policies, inaccordance with an embodiment;

FIG. 11 is a schematic drawing of a graphical user interface fordetermining demographic information of a household, in accordance withan embodiment;

FIG. 12 is an schematic drawing illustrating an example of apattern-based policy implementation, in accordance with an embodiment;

FIG. 13 is a schematic drawing illustrating a graphical user interfacefor manually constructing household policies, in accordance with anembodiment;

FIG. 14 is a flow diagram illustrating a process for providing sensorconfidence related to a particular policy, in accordance with anembodiment;

FIG. 15 is a flow diagram illustrating a process for presenting sensorconfidence for a new policy, in accordance with an embodiment;

FIG. 16 is a flow diagram illustrating a process for suggestingmodifications to enhance sensor confidence, in accordance with anembodiment;

FIG. 17 is a schematic drawing illustrating examples of outputs from theprocess of FIG. 16, in accordance with an embodiment;

FIG. 18 is a flow diagram illustrating a process making adjustmentsbased upon manual modifications of a policy, in accordance with anembodiment;

FIG. 19 is a flow diagram illustrating a process for monitoringactivities of a household, in accordance with an embodiment;

FIG. 20 is a schematic drawing illustrating policies implementedaccording to locational zones, in accordance with an embodiment;

FIG. 21 is a schematic drawing of a system for monitoring dining, inaccordance with an embodiment;

FIG. 22 is a flow diagram illustrating a process for monitoring dining,in accordance with an embodiment;

FIG. 23 is a schematic drawing of a system for monitoring activities, inaccordance with an embodiment;

FIGS. 24 and 25 are flow diagrams illustrating processes for monitoringmedia consumption, in accordance with an embodiment;

FIG. 26 is a schematic drawing of a system for monitoring officeactivities, in accordance with an embodiment;

FIG. 27 is a flow diagram illustrating a process for monitoring officeactivities, in accordance with an embodiment;

FIG. 28 is a schematic drawing of a system for monitoring kitchenactivities, in accordance with an embodiment;

FIG. 29 is a flow diagram illustrating a process for monitoring kitchenactivities, in accordance with an embodiment;

FIG. 30 is a schematic drawing of a system for monitoring bathroomactivities while maintaining personal privacy, in accordance with anembodiment;

FIGS. 31-33 are flow diagram illustrating processes for monitoringbathroom activities while maintaining personal privacy, in accordancewith an embodiment;

FIG. 34 is a flow diagram illustrating a process for reporting compiledactivities, in accordance with an embodiment;

FIG. 35 is a schematic drawing of a graphical user interface useful forreporting compiled activities, in accordance with an embodiment;

FIG. 36 is a flow diagram illustrating a process for detecting childmischief, in accordance with an embodiment;

FIG. 37 is a flow diagram illustrating a process for detecting emotions,in accordance with an embodiment;

FIG. 38 is a flow diagram illustrating a process for detectingundesirable actions, in accordance with an embodiment;

FIG. 39 is a flow diagram illustrating a process for detecting access toundesirable substances, in accordance with an embodiment;

FIG. 40 is a flow diagram illustrating a process for detecting chorecompletion status, in accordance with an embodiment;

FIG. 41 is a flow diagram illustrating a process for monitoring medicalsymptoms, in accordance with an embodiment;

FIG. 42 is a flow diagram illustrating a process for situationalmonitoring, in accordance with an embodiment;

FIG. 43 is a flow diagram illustrating a process for situationalmonitoring of children who are home alone, in accordance with anembodiment;

FIG. 44 is a flow diagram illustrating a process for situationalmonitoring of non-parental supervision, in accordance with anembodiment;

FIGS. 45 and 46 are flow diagrams illustrating processes for monitoringassertion activities, in accordance with an embodiment;

FIG. 47 is a flow diagram illustrating a process for providing controlof a smart-device environment, in accordance with an embodiment;

FIG. 48 is a flow diagram illustrating a process for situation-basedcontrol, in accordance with an embodiment;

FIG. 49 is a flow diagram illustrating a process for controlling asmart-device environment under non-parental supervision, in accordancewith an embodiment;

FIG. 50 is a flow diagram illustrating a process for grounding controlof a smart-device environment, in accordance with an embodiment;

FIG. 51 is a flow diagram illustrating a process for reward-basedcontrol of a smart-device environment, in accordance with an embodiment;

FIG. 52 is a schematic drawing illustrating allotment-based control of asmart-device environment, in accordance with an embodiment;

FIG. 53 is a flow diagram illustrating a process for the allotment-basedcontrol of the smart-device environment, in accordance with anembodiment;

FIG. 54 is a flow diagram illustrating a process for progressive controlin a smart-device environment, in accordance with an embodiment;

FIG. 55 is a data chart illustrating control via a progressivelyincreasing control threshold, in accordance with an embodiment;

FIG. 56 is a schematic drawing illustrating control via progressivelyincreasing thresholds, in accordance with an embodiment;

FIG. 57 is a data chart illustrating control via a progressivelydecreasing control threshold, in accordance with an embodiment; and

FIG. 58 is a schematic drawing illustrating control via progressivelydecreasing thresholds, in accordance with an embodiment.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

One or more specific embodiments will be described below. In an effortto provide a concise description of these embodiments, not all featuresof an actual implementation are described in the specification. Itshould be appreciated that in the development of any such actualimplementation, as in any engineering or design project, numerousimplementation-specific decisions must be made to achieve thedevelopers' specific goals, such as compliance with system-related andbusiness-related constraints, which may vary from one implementation toanother. Moreover, it should be appreciated that such a developmenteffort might be complex and time consuming, but would nevertheless be aroutine undertaking of design, fabrication, and manufacture for those ofordinary skill having the benefit of this disclosure.

Embodiments of the present disclosure relate to smart-deviceenvironments. In particular, a smart-device environment may be providedwith and/or create one or more environment policies. These policies maybe used to report characteristics, attributes, etc. of the environmentand/or occupants of the environment. Further, these policies may be usedto facilitate control of one or more smart-devices within theenvironment, based upon one or more conditions defined in the policies.

Smart Device in Smart Home Environment

By way of introduction, FIG. 1 illustrates an example of a generaldevice 10 that may that may be disposed within a building environment.In one embodiment, the device 10 may include one or more sensors 12, auser-interface component 14, a power supply 16 (e.g., including a powerconnection and/or battery), a network interface 18 (which may be wiredand/or wireless, and if wireless may comprise a single radio accordingto a single wireless protocol or multiple radios according to multipledifferent wireless protocols), a high-power processor 20, a low-powerprocessor 22, a passive infrared (PIR) sensor 24, a light source 26, andthe like.

The sensors 12, in certain embodiments, may detect various propertiessuch as acceleration, temperature, humidity, water, supplied power,proximity, external motion, device motion, sound signals, ultrasoundsignals, light signals, fire, smoke, carbon monoxide or other gas,global-positioning-satellite (GPS) signals, radio-frequency (RF), otherelectromagnetic signals or fields, or the like. As such, the sensors 12may include temperature sensor(s), humidity sensor(s), hazard-relatedsensor(s) or other environmental sensor(s), accelerometer(s),microphone(s), optical sensors up to and including camera(s) (e.g.,charged coupled-device or video cameras), active or passive radiationsensors, GPS receiver(s) or radiofrequency identification detector(s).While FIG. 1 illustrates an embodiment with two sensor blocks, manyembodiments may include as few as a single sensor, or may includemultiple sensor blocks each containing one or more sensors, up to andincluding multiple blocks each containing entire arrays of sensors. Insome instances, the device 10 may include one or more primary sensorsand one or more secondary sensors. Here, the primary sensor(s) may sensedata central to the core operation of the device (e.g., sensing atemperature in a thermostat or sensing smoke in a smoke detector), whilethe secondary sensor(s) may sense other types of data (e.g., motion,light or sound), which can be used for energy-efficiency objectives,smart-operation objectives, individual-component objectives as part of alarger home automation system or functionality, or any of a variety ofother objectives.

One or more user-interface components 14 in the device 10 may receiveinput from the user and/or present information to the user. The receivedinput may be used to determine a setting. In certain embodiments, theuser-interface components may include a mechanical or virtual componentthat responds to the user's motion. For example, the user canmechanically move a sliding component (e.g., along a vertical orhorizontal track) or rotate a rotatable ring (e.g., along a circulartrack), or the user's motion along a touchpad may be detected. Suchmotions may correspond to a setting adjustment, which can be determinedbased on an absolute position of a user-interface component 14 or basedon a displacement of a user-interface components 14 (e.g., adjusting aset point temperature by 1 degree F. for every 10° rotation of arotatable-ring component). Physically and virtually movableuser-interface components can allow a user to set a setting along aportion of an apparent continuum. Thus, the user may not be confined tochoose between two discrete options (e.g., as would be the case if upand down buttons were used) but can quickly and intuitively define asetting along a range of possible setting values. For example, amagnitude of a movement of a user-interface component may be associatedwith a magnitude of a setting adjustment, such that a user maydramatically alter a setting with a large movement or finely tune asetting with a small movement.

The user-interface components 14 may also include one or more buttons(e.g., up and down buttons), a keypad, a number pad, a switch, amicrophone, and/or a camera (e.g., to detect gestures). In oneembodiment, the user-interface component 14 may include aclick-and-rotate annular ring component that may enable the user tointeract with the component by rotating the ring (e.g., to adjust asetting) and/or by clicking the ring inwards (e.g., to select anadjusted setting or to select an option). In another embodiment, theuser-interface component 14 may include a camera that may detectgestures (e.g., to indicate that a power or alarm state of a device isto be changed). In some instances, the device 10 may have one primaryinput component, which may be used to set a plurality of types ofsettings. The user-interface components 14 may also be configured topresent information to a user via, e.g., a visual display (e.g., athin-film-transistor display or organic light-emitting-diode display)and/or an audio speaker.

The power-supply component 16 may include a power connection and/or alocal battery. For example, the power connection may connect the device10 to a power source such as a line voltage source and/or an on-boardpower generating component, such as solar power generation components,other power harvesting components, or the like. In some instances, an ACpower source can be used to repeatedly charge a (e.g., rechargeable)local battery, such that the battery may be used later to supply powerto the device 10 when the AC power source is not available, or tosupplement power to the device 10 when instantaneous power requirementsexceed that which can be provided by the AC power source or by local ACpower conversion circuitry.

The network interface 18 may include a component that enables the device10 to communicate between devices. As such, the network interface 18 mayenable the device 10 to communicate with other devices 10 via a wired orwireless network. The network interface 18 may include a wireless cardor some other transceiver connection to facilitate this communication.The network interface 18 may incorporate one or more of a plurality ofstandard or proprietary protocols including, but not limited to, Wi-Fi,Zigbee, Z-Wave, or Thread. One particularly advantageous protocol forimplementation by the network interface 18 is the Thread protocol,promulgated by the Thread Group and including features of 802.15.4, IETFIPv6, and 6LoWPAN protocols.

The high-power processor 20 and the low-power processor 22 may supportone or more of a variety of different device functionalities. As such,the high-power processor 20 and the low-power processor 22 may eachinclude one or more processors configured and programmed to carry outand/or cause to be carried out one or more of the functionalitiesdescribed herein. In one embodiment, the high-power processor 20 and thelow-power processor 22 may include general-purpose processors carryingout computer code stored in local memory (e.g., flash memory, harddrive, random access memory), special-purpose processors orapplication-specific integrated circuits, combinations thereof, and/orusing other types of hardware/firmware/software processing platforms. Incertain embodiments, the high-power processor 20 may executecomputationally intensive operations such as operating theuser-interface component 14 and the like. The low-power processor 22, onthe other hand, may manage less complex processes such as detecting ahazard or temperature from the sensor 12. In one embodiment that isparticularly advantageous for battery-only or other power-constrainedenvironments, the high-power processor 20 may be maintained in a sleepstate most of the time, and the low-power processor may be used to wakeor initialize the high-power processor for intervals whencomputationally intensive processes are required.

By way of example, the high-power processor 20 and the low-powerprocessor 22 may detect when a location (e.g., a house or room) isoccupied (i.e., includes a presence of a human), up to and includingwhether it is occupied by a specific person or is occupied by a specificnumber of people (e.g., relative to one or more thresholds). In oneembodiment, this detection can occur, e.g., by analyzing microphonesignals, detecting user movements (e.g., in front of a device),detecting openings and closings of doors or garage doors, detectingwireless signals, detecting an internet protocol (IP) address of areceived signal, detecting operation of one or more devices within atime window, or the like. Moreover, the high-power processor 20 and thelow-power processor 22 may include image recognition technology toidentify particular occupants or objects.

In certain embodiments, the high-power processor 20 and the low-powerprocessor 22 may detect the presence of a human using the PIR sensor 24.The PIR sensor 24 may be a passive infrared sensor that measuresinfrared (IR) radiation emitted from or reflected by objects in itsfield of view.

In some instances, the high-power processor 20 may predict desirablesettings and/or implement those settings. For example, based on thepresence detection, the high-power processor 20 may adjust devicesettings to, e.g., conserve power when nobody is home or in a particularroom or to accord with user preferences (e.g., general at-homepreferences or user-specific preferences). As another example, based onthe detection of a particular person, animal or object (e.g., a child,pet or lost object), the high-power processor 20 may initiate an audioor visual indicator of where the person, animal or object is or mayinitiate an alarm or security feature if an unrecognized person isdetected under certain conditions (e.g., at night or when lights areoff).

In some embodiments, multiple instances of the device 10 (which may besimilar to or different than each other) may interact with each othersuch that events detected by a first device influences actions of asecond device. For example, a first device can detect that a user hasentered into a garage (e.g., by detecting motion in the garage,detecting a change in light in the garage or detecting opening of thegarage door). The first device can transmit this information to a seconddevice via the network interface 18, such that the second device can,e.g., adjust a home temperature setting, a light setting, a musicsetting, and/or a security-alarm setting. As another example, a firstdevice can detect a user approaching a front door (e.g., by detectingmotion or sudden light pattern changes). The first device may, e.g.,cause a general audio or visual signal to be presented (e.g., such assounding of a doorbell) or cause a location-specific audio or visualsignal to be presented (e.g., to announce the visitor's presence withina room that a user is occupying).

In addition to detecting various types of events, the device 10 mayinclude a light source 26 that may illuminate when a living being, suchas a human, is detected as approaching. The light source 26 may includeany type of light source such as one or more light-emitting diodes orthe like. The light source 26 may be communicatively coupled to thehigh-power processor 20 and the low-power processor 22, which mayprovide a signal to cause the light source 26 to illuminate.

Keeping the foregoing in mind, FIG. 2 illustrates an example of asmart-device environment 30 within which one or more of the devices 10of FIG. 1 and within which one or more of the methods, systems,services, and/or computer program products described further herein canbe applicable. The depicted smart-device environment 30 includes astructure 32, which can include, e.g., a house, office building, garage,or mobile home. It will be appreciated that devices can also beintegrated into a smart-device environment 30 that does not include anentire structure 32, such as an apartment, condominium, or office space.Further, the smart home environment can control and/or be coupled todevices outside of the actual structure 32. Indeed, several devices inthe smart home environment need not physically be within the structure32 at all. For example, a device controlling a pool heater or irrigationsystem can be located outside of the structure 32.

The depicted structure 32 includes a plurality of rooms 38, separated atleast partly from each other via walls 40. The walls 40 can includeinterior walls or exterior walls. Each room can further include a floor42 and a ceiling 44. Devices can be mounted on, integrated with, and/orsupported by a wall 40, floor 42 or ceiling 44.

In some embodiments, the smart-device environment 30 of FIG. 2 includesa plurality of devices 10, including intelligent, multi-sensing,network-connected devices, that can integrate seamlessly with each otherand/or with a central server or a cloud-computing system to provide anyof a variety of useful smart-home objectives. The smart-deviceenvironment 30 may include one or more intelligent, multi-sensing,network-connected thermostats 46 (hereinafter referred to as “smartthermostats 46”), one or more intelligent, network-connected,multi-sensing hazard detection units 50 (hereinafter referred to as“smart hazard detectors 50”), and one or more intelligent,multi-sensing, network-connected entryway interface devices 52(hereinafter referred to as “smart doorbells 52”). According toembodiments, the smart thermostat 46 may include a Nest LearningThermostat from Nest Labs, Inc., among others. Alone or in combinationwith each other and/or with one or more other temperature and/orhumidity sensors of the smart home, the smart thermostats 46 detectambient climate characteristics (e.g., temperature and/or humidity) andcontrol an HVAC system 48 accordingly.

The smart hazard detector 50 may detect the presence of a hazardoussubstance or a substance indicative of a hazardous substance (e.g.,smoke, fire, or carbon monoxide). The smart hazard detector 50 mayinclude a Nest Protect Smoke+CO Alarm that may include sensors 12 suchas smoke sensors, carbon monoxide sensors, and the like. As such, thehazard detector 50 may determine when smoke, fire, or carbon monoxidemay be present within the building.

The smart doorbell 52 may detect a person's approach to or departurefrom a location (e.g., an outer door), provide doorbell functionality,announce a person's approach or departure via audio or visual means, orcontrol settings on a security system (e.g., to activate or deactivatethe security system when occupants go and come). The smart doorbell 52may interact with other devices 10 based on whether someone hasapproached or entered the smart-device environment 30.

In some embodiments, the smart-device environment 30 further includesone or more intelligent, multi-sensing, network-connected wall switches54 (hereinafter referred to as “smart wall switches 54”), along with oneor more intelligent, multi-sensing, network-connected wall pluginterfaces 56 (hereinafter referred to as “smart wall plugs 56”). Thesmart wall switches 54 may detect ambient lighting conditions, detectroom-occupancy states, and control an ON/OFF or dimming state of one ormore lights. In some instances, smart wall switches 54 may also controla power state or speed of a fan, such as a ceiling fan. The smart wallplugs 56 may detect occupancy of a room or enclosure and control supplyof power to one or more wall plugs (e.g., such that power is notsupplied to the plug if nobody is at home).

Still further, in some embodiments, the device 10 within thesmart-device environment 30 may further includes a plurality ofintelligent, multi-sensing, network-connected appliances 58 (hereinafterreferred to as “smart appliances 58”), such as refrigerators, stovesand/or ovens, televisions, washers, dryers, lights, stereos, intercomsystems, garage-door openers, floor fans, ceiling fans, wall airconditioners, pool heaters, irrigation systems, security systems,cameras 59 (e.g., Nest Dropcams) and so forth. According to embodiments,the network-connected appliances 58 are made compatible with thesmart-device environment by cooperating with the respectivemanufacturers of the appliances. For example, the appliances can bespace heaters, window AC units, motorized duct vents, etc. When pluggedin, an appliance can announce itself to the smart-home network, such asby indicating what type of appliance it is, and it can automaticallyintegrate with the controls of the smart-home. Such communication by theappliance to the smart home can be facilitated by any wired or wirelesscommunication protocols known by those having ordinary skill in the art.The smart home also can include a variety of non-communicating legacyappliances 68, such as old conventional washer/dryers, refrigerators,and the like which can be controlled, albeit coarsely (ON/OFF), byvirtue of the smart wall plugs 56. The smart-device environment 30 canfurther include a variety of partially communicating legacy appliances70, such as infrared (“IR”) controlled wall air conditioners or otherIR-controlled devices, which can be controlled by IR signals provided bythe smart hazard detectors 50 or the smart wall switches 54.

According to embodiments, the smart thermostats 46, the smart hazarddetectors 50, the smart doorbells 52, the smart wall switches 54, thesmart wall plugs 56, and other devices of the smart-device environment30 are modular and can be incorporated into older and new houses. Forexample, the devices 10 are designed around a modular platformconsisting of two basic components: a head unit and a back plate, whichis also referred to as a docking station. Multiple configurations of thedocking station are provided so as to be compatible with any home, suchas older and newer homes. However, all of the docking stations include astandard head-connection arrangement, such that any head unit can beremovably attached to any docking station. Thus, in some embodiments,the docking stations are interfaces that serve as physical connectionsto the structure and the voltage wiring of the homes, and theinterchangeable head units contain all of the sensors 12, processors 28,user interfaces 14, the power supply 16, the network interface 18, andother functional components of the devices described above.

The smart-device environment 30 may also include communication withdevices outside of the physical home but within a proximate geographicalrange of the home. For example, the smart-device environment 30 mayinclude a pool heater controller 34 that communicates a current pooltemperature to other devices within the smart-device environment 30 orreceives commands for controlling the pool temperature. Similarly, thesmart-device environment 30 may include an irrigation controller 36 thatcommunicates information regarding irrigation systems within thesmart-device environment 30 and/or receives control information forcontrolling such irrigation systems. According to embodiments, analgorithm is provided for considering the geographic location of thesmart-device environment 30, such as based on the zip code or geographiccoordinates of the home. The geographic information is then used toobtain data helpful for determining optimal times for watering, suchdata may include sun location information, temperature, dewpoint, soiltype of the land on which the home is located, etc.

By virtue of network connectivity, one or more of the smart-home devicesof FIG. 2 can further allow a user to interact with the device even ifthe user is not proximate to the device. For example, a user cancommunicate with a device using a computer (e.g., a desktop computer,laptop computer, or tablet) or other portable electronic device (e.g., asmartphone) 66. A web page or app can be configured to receivecommunications from the user and control the device based on thecommunications and/or to present information about the device'soperation to the user. For example, the user can view a current setpointtemperature for a device and adjust it using a computer. The user can bein the structure during this remote communication or outside thestructure.

As discussed, users can control the smart thermostat and other smartdevices in the smart-device environment 30 using a network-connectedcomputer or portable electronic device 66. In some examples, some or allof the occupants (e.g., individuals who live in the home) can registertheir device 66 with the smart-device environment 30. Such registrationcan be made at a central server to authenticate the occupant and/or thedevice as being associated with the home and to give permission to theoccupant to use the device to control the smart devices in the home. Anoccupant can use their registered device 66 to remotely control thesmart devices of the home, such as when the occupant is at work or onvacation. The occupant may also use their registered device to controlthe smart devices when the occupant is actually located inside the home,such as when the occupant is sitting on a couch inside the home. Itshould be appreciated that instead of or in addition to registeringdevices 66, the smart-device environment 30 may make inferences aboutwhich individuals live in the home and are therefore occupants and whichdevices 66 are associated with those individuals. As such, thesmart-device environment “learns” who is an occupant and permits thedevices 66 associated with those individuals to control the smartdevices of the home.

In some instances, guests desire to control the smart devices. Forexample, the smart-device environment may receive communication from anunregistered mobile device of an individual inside of the home, wheresaid individual is not recognized as an occupant of the home. Further,for example, a smart-device environment may receive communication from amobile device of an individual who is known to be, or who is registeredas, a guest.

According to embodiments, a guest-layer of controls can be provided toguests of the smart-device environment 30. The guest-layer of controlsgives guests access to basic controls (e.g., a judicially selectedsubset of features of the smart devices), such as temperatureadjustments, but it locks out other functionalities. The guest layer ofcontrols can be thought of as a “safe sandbox” in which guests havelimited controls, but they do not have access to more advanced controlsthat could fundamentally alter, undermine, damage, or otherwise impairthe occupant-desired operation of the smart devices. For example, theguest layer of controls will not permit the guest to adjust theheat-pump lockout temperature.

As described below, the smart thermostat 46 and other smart devices“learn” by observing occupant behavior. For example, the smartthermostat learns occupants' preferred temperature set-points formornings and evenings, and it learns when the occupants are asleep orawake, as well as when the occupants are typically away or at home, forexample. According to embodiments, when a guest controls the smartdevices, such as the smart thermostat, the smart devices do not “learn”from the guest. This prevents the guest's adjustments and controls fromaffecting the learned preferences of the occupants.

According to some embodiments, a smart television remote control isprovided. The smart remote control recognizes occupants by thumbprint,visual identification, RFID, etc., and it recognizes a user as a guestor as someone belonging to a particular class having limited control andaccess (e.g., child). Upon recognizing the user as a guest or someonebelonging to a limited class, the smart remote control only permits thatuser to view a subset of channels and to make limited adjustments to thesettings of the television and other devices. For example, a guestcannot adjust the digital video recorder (DVR) settings, and a child islimited to viewing child-appropriate programming.

According to some embodiments, similar controls are provided for otherinstruments, utilities, and devices in the house. For example, sinks,bathtubs, and showers can be controlled by smart spigots that recognizeusers as guests or as children and therefore prevent water fromexceeding a designated temperature that is considered safe.

In some embodiments, in addition to containing processing and sensingcapabilities, each of the devices 34, 36, 46, 50, 52, 54, 56, and 58(collectively referred to as “the smart devices”) is capable of datacommunications and information sharing with any other of the smartdevices, as well as to any central server or cloud-computing system orany other device that is network-connected anywhere in the world. Therequired data communications can be carried out using any of a varietyof custom or standard wireless protocols (Wi-Fi, ZigBee, 6LoWPAN, etc.)and/or any of a variety of custom or standard wired protocols (CAT6Ethernet, HomePlug, etc.).

According to embodiments, all or some of the smart devices can serve aswireless or wired repeaters, and the smart devices can be configured toform a mesh network for reliable, robust, fault-tolerantdevice-to-device and device-to-cloud communications. For example, afirst one of the smart devices can communicate with a second one of thesmart device via a wireless router 60. The smart devices can furthercommunicate with each other via a connection to a network, such as theInternet 62. Through the Internet 62, the smart devices can communicatewith a central server or a cloud-computing system 64. The central serveror cloud-computing system 64 can be associated with a manufacturer,support entity, or service provider associated with the device. For oneembodiment, a user may be able to contact customer support using adevice itself rather than needing to use other communication means suchas a telephone or Internet-connected computer. Further, software updatescan be automatically sent from the central server or cloud-computingsystem 64 to devices (e.g., when available, when purchased, or atroutine intervals). According to embodiments, as described further inthe commonly assigned WO2014047501A1, which is incorporated by referenceherein, the smart devices may combine to create a mesh network ofspokesman and low-power nodes in the smart-device environment 30, wheresome of the smart devices are “spokesman” nodes and others are“low-powered” nodes.

An example of a low-power node is a smart night light 65. In addition tohousing a light source, the smart night light 65 houses an occupancysensor, such as an ultrasonic or passive IR sensor, and an ambient lightsensor, such as a photoresistor or a single-pixel sensor that measureslight in the room. In some embodiments, the smart night light 65 isconfigured to activate the light source when its ambient light sensordetects that the room is dark and when its occupancy sensor detects thatsomeone is in the room. In other embodiments, the smart night light 65is simply configured to activate the light source when its ambient lightsensor detects that the room is dark. Further, according to embodiments,the smart night light 65 includes a low-power wireless communicationchip (e.g., ZigBee chip) that regularly sends out messages regarding theoccupancy of the room and the amount of light in the room, includinginstantaneous messages coincident with the occupancy sensor detectingthe presence of a person in the room. As mentioned above, these messagesmay be sent wirelessly, using the mesh network, from node to node (i.e.,smart device to smart device) within the smart-device environment 30 aswell as over the Internet 62 to the central server or cloud-computingsystem 64.

In some embodiments, the low-powered and spokesman nodes (e.g., devices46, 50, 52, 54, 56, 58, and 65) can function as “tripwires” for an alarmsystem in the smart-device environment. For example, in the event aperpetrator circumvents detection by alarm sensors located at windows,doors, and other entry points of the smart-device environment 30, thealarm could be triggered upon receiving an occupancy, motion, heat,sound, etc. message from one or more of the low-powered and spokesmannodes in the mesh network. For example, upon receiving a message from asmart night light 65 indicating the presence of a person, the centralserver or cloud-computing system 64 or some other device could triggeran alarm, provided the alarm is armed at the time of detection. Thus,the alarm system could be enhanced by various low-powered and spokesmannodes located throughout the smart-device environment 30. In thisexample, a user could enhance the security of the smart-deviceenvironment 30 by buying and installing extra smart nightlights 65.

Further included and illustrated in the smart-device environment 30 ofFIG. 2 are service robots 69 each configured to carry out, in anautonomous manner, any of a variety of household tasks. For someembodiments, the service robots 69 can be respectively configured toperform floor sweeping, floor washing, etc. in a manner similar to thatof known commercially available devices such as the ROOMBA™ and SCOOBA™products sold by iRobot, Inc. of Bedford, Mass.

FIG. 3 illustrates a network-level view of an extensible devices andservices platform 80 with which the smart-home environment of FIG. 2 canbe integrated, according to an embodiment. The extensible devices andservices platform 80 can be concentrated at a single server ordistributed among several different computing entities withoutlimitation with respect to the smart-device environment 30. Theextensible devices and services platform 80 may include a processingengine 86, which may include engines that receive data from devices ofsmart-device environments (e.g., via the Internet or a hubbed network),to index the data, to analyze the data and/or to generate statisticsbased on the analysis or as part of the analysis. The analyzed data canbe stored as derived home data 88.

Results of the analysis or statistics can thereafter be transmitted backto the device that provided home data used to derive the results, toother devices, to a server providing a web page to a user of the device,or to other non-device entities. For example, use statistics, usestatistics relative to use of other devices, use patterns, and/orstatistics summarizing sensor readings can be generated by theprocessing engine 86 and transmitted. The results or statistics can beprovided via the Internet 62. In this manner, the processing engine 86can be configured and programmed to derive a variety of usefulinformation from the home data 82. A single server can include one ormore engines.

The derived data can be highly beneficial at a variety of differentgranularities for a variety of useful purposes, ranging from explicitprogrammed control of the devices on a per-home, per-neighborhood, orper-region basis (for example, demand-response programs for electricalutilities), to the generation of inferential abstractions that canassist on a per-home basis (for example, an inference can be drawn thatthe homeowner has left for vacation and so security detection equipmentcan be put on heightened sensitivity), to the generation of statisticsand associated inferential abstractions that can be used for governmentor charitable purposes. For example, processing engine 86 can generatestatistics about device usage across a population of devices and sendthe statistics to device users, service providers or other entities(e.g., that have requested or may have provided monetary compensationfor the statistics).

FIG. 4 illustrates an abstracted functional view 110 of the extensibledevices and services platform 80 of FIG. 3, with particular reference tothe processing engine 86 as well as devices, such as those of thesmart-device environment 30 of FIG. 2. Even though devices situated insmart-device environments will have an endless variety of differentindividual capabilities and limitations, they can all be thought of assharing common characteristics in that each of them is a data consumer112 (DC), a data source 114 (DS), a services consumer 116 (SC), and aservices source 118 (SS). Advantageously, in addition to providing theessential control information needed for the devices to achieve theirlocal and immediate objectives, the extensible devices and servicesplatform 80 can also be configured to harness the large amount of datathat is flowing out of these devices. In addition to enhancing oroptimizing the actual operation of the devices themselves with respectto their immediate functions, the extensible devices and servicesplatform 80 can be directed to “repurposing” that data in a variety ofautomated, extensible, flexible, and/or scalable ways to achieve avariety of useful objectives. These objectives may be predefined oradaptively identified based on, e.g., usage patterns, device efficiency,and/or user input (e.g., requesting specific functionality).

For example, FIG. 4 shows processing engine 86 as including a number ofparadigms 120. Processing engine 86 can include a managed servicesparadigm 120 a that monitors and manages primary or secondary devicefunctions. The device functions can include ensuring proper operation ofa device given user inputs, estimating that (e.g., and responding to aninstance in which) an intruder is or is attempting to be in a dwelling,detecting a failure of equipment coupled to the device (e.g., a lightbulb having burned out), implementing or otherwise responding to energydemand response events, or alerting a user of a current or predictedfuture event or characteristic. Processing engine 86 can further includean advertising/communication paradigm 120 b that estimatescharacteristics (e.g., demographic information), desires and/or productsof interest of a user based on device usage. Services, promotions,products or upgrades can then be offered or automatically provided tothe user. Processing engine 86 can further include a social paradigm 120c that uses information from a social network, provides information to asocial network (for example, based on device usage), and/or processesdata associated with user and/or device interactions with the socialnetwork platform. For example, a user's status as reported to theirtrusted contacts on the social network could be updated to indicate whenthey are home based on light detection, security system inactivation ordevice usage detectors. As another example, a user may be able to sharedevice-usage statistics with other users. In yet another example, a usermay share HVAC settings that result in low power bills and other usersmay download the HVAC settings to their smart thermostat 46 to reducetheir power bills.

The processing engine 86 can include achallenges/rules/compliance/rewards paradigm 120 d that informs a userof challenges, competitions, rules, compliance regulations and/orrewards and/or that uses operation data to determine whether a challengehas been met, a rule or regulation has been complied with and/or areward has been earned. The challenges, rules or regulations can relateto efforts to conserve energy, to live safely (e.g., reducing exposureto toxins or carcinogens), to conserve money and/or equipment life, toimprove health, etc. For example, one challenge may involve participantsturning down their thermostat by one degree for one week. Those thatsuccessfully complete the challenge are rewarded, such as by coupons,virtual currency, status, etc. Regarding compliance, an example involvesa rental-property owner making a rule that no renters are permitted toaccess certain owner's rooms. The devices in the room having occupancysensors could send updates to the owner when the room is accessed.

The processing engine 86 can integrate or otherwise utilize extrinsicinformation 122 from extrinsic sources to improve the functioning of oneor more processing paradigms. Extrinsic information 122 can be used tointerpret data received from a device, to determine a characteristic ofthe environment near the device (e.g., outside a structure that thedevice is enclosed in), to determine services or products available tothe user, to identify a social network or social-network information, todetermine contact information of entities (e.g., public-service entitiessuch as an emergency-response team, the police or a hospital) near thedevice, etc., to identify statistical or environmental conditions,trends or other information associated with a home or neighborhood, andso forth.

Household Policy Provision

Having now discussed the smart-device or smart-home environment, thediscussion now turns to sourcing policy that may be executed by smartdevices within the smart-device environment. In some embodiments suchexecution can be carried out by individual smart devices acting alone,while in other embodiments such execution can be carried out byself-orchestrating groupings (static or dynamic) of smart devices actingin concert, while in still other embodiments such execution can becarried out by individual smart devices or groups of smart devicesacting under the orchestration of a central server, such as acloud-based computing system located remotely from the home. FIG. 5 is aschematic drawing of an environment 150 of household policy equippedhomes 152, in accordance with an embodiment. Policies may reachhousehold policy equipped homes 152 in a variety of manners. Forexample, a household policy manager 154 may be responsible for supplyingpolicies to homes 152 based upon: household goals, household-definedpolicies, smart-device capabilities within the home 152, shared policiesamong linked homes, household demographics, etc. While FIG. 5illustrates a household policy manager 154 external to each of the homes152, the household policy manager 154 may be local to each house 152, ormay work in conjunction with other household policy managers 154 thatare all local to homes 152, all remote from the homes 152, or are acombination of local and remote policy managers 154. For example, home156 includes a local household policy manager 154 that may interact witha remote household policy manager 154.

In the illustrated embodiment, Home 156 is set up to receive goal-basedpolicies. For example, as will be discussed in more detail below, a goalmight be to “Spend less time on electronic devices” or “Use 5% lessenergy each month for the next 3 months.” The household goals 158 areprovided to the household policy manager 154, which supplies goal-basedpolicies 160 for execution within the home 156. In some embodiments, thehousehold goals 158 may be obtained by providing a survey to one or morehousehold members (e.g., via a graphical user interface). In someembodiments, household goals may be suggested by a computer (e.g., thehousehold policy manager 154) based upon observed behaviors of membersof the household, based upon certain inputs provided from smart devicesin the home, based upon certain inputs from remotevendors/facilitators/regulators/etc., or any combination thereof. Thesesuggested household goals may result in particular selected householdgoals 158.

FIG. 5 illustrates that Home 162 is set up to receive user-definedpolicies 164. For example, there may be a graphical user interfacewithin home 162 that enables members of the household to create policiespersonalized for the home 162. In some embodiments, these policies maybe shared with a household policy manager 154 communicatively coupledwith a plurality of policy equipped homes 152, such that theuser-defined policies 164 may be shared with other policy equipped homes152. For example, in the current embodiment, the user-defined policies164 are provided to the household policy manager 154 with householddemographic information 166 (e.g., number of household members, ages ofhousehold members, activity patterns of household members, etc.). Byreceiving both the user-defined policies 164 and the demographicinformation 166, the household policy manager 154 may provide/suggestthe user-defined policies 164 to other policy equipped homes 152 havingsimilar demographic information 166.

In some embodiments, policies may be provided for execution at thepolicy equipped homes 152 based upon capabilities of devices within thehome 152. For example, home 168 is set up to receive policies based uponcapabilities within the home 168. The capabilities 170, such as: “thishome is able to detect occupancy at a dining room table” or “this homeis able to hear audio in the living room,” are provided to the householdpolicy manager 154. The household policy manager 154 may determine whichof a plurality of available policies are available for implementation atthe home 168 and provide this list of capability-based policies 172 tothe home 168. A policy administrator within the home 168 may selectpolicies from the capability-based polices 172 to implement within thehome 168.

In some embodiments, household policies may be linked. In other words,there may be a relationship between two policy equipped homes 152resulting in execution of linked rules within at least one of the policyequipped homes 152. For example, in a situation where there is a sharedcustody agreement regarding children between two policy equipped homes152, household policies may be transferred from one home 152 to anotherhome 152, such that the children's policies are implemented at eitherhome. Further, in some embodiments, the linking may include amaster-slave arrangement, where one house 152 controls the implementedpolicies of another house 152. For example, in one example, an adultchild may be tasked with caring for an elderly parent in another house152. In this example, the adult child may have authority over thepolicies implemented at the elderly parent's home 152. For example, theadult child may have authority to cause execution at the elderlyparent's house a policy for presenting a reminder to take medications ata particular time of day. In the current example, home 174 is linkedwith home 176. Linked policies 178 to be implemented at the respectivelinked home are provided to the household policy manager 154, where theyare provided as policies 180 to be implemented in the linkedenvironment.

As discussed above, in some embodiments, household demographics may beused to determine particular policies for a household. For example,different policies may be desirable for a house with small children thana household of all adults. Demographic information may include, forexample: occupant information such as: number of occupants, gender ofoccupants, age of occupants, ethnicity of occupants, etc. Thedemographic information 166 may be provided to the household policymanager 154, which may provide policies 182 based upon the demographics.For example, as mentioned above, user-defined policies 164 associatedwith matching demographics 166 may be provided to house 184.

Additionally, household behavioral patterns 186 may be used to providepolicies. Observed behavioral patterns 186 may be provided to thehousehold policy manager 154. The household policy manager 154 maydetermine policies that correspond to the behavioral patterns andprovide suggested policies 188 based on the behaviors 186. For example,one policy may arise from an observation that a household occupantattempts to reach their place of employment at 8:00 AM each work day.Based upon this detected behavioral pattern 186, a policy may beprovided to dynamically adjust an alarm clock wait time based upontraffic delays, etc.

Any number of policy provision schemes may be used alone or incombination to provide policies to a household policy equipped home 152.For example, while home 156 is illustrated as being provided withpolicies based upon household goals 158, the provided policies couldfurther be limited and/or supplemented based upon other criteria (e.g.,limited based upon capabilities 170 of home 156 and/or supplemented bybehavior-based policies 188 of the home 156).

Further, policy provisioning and implementation may be provided basedupon varying degrees of efficiency versus convenience and/or varyingdegrees of leniency. For example, some users may wish to tightly controlhousehold policy aspects while others would rather loosely controlpolicy implementation. In some embodiments, the household policy manager154 may enable a user to confirm and/or set each aspect of a policyprior to implementation. Alternatively, the household policy manager 154may enable automatic policy implementation without user control and/orconfirmation. In some embodiments, a hybrid override mode may beimplemented that enables the user to halt implementation of anautomatically implemented policy and/or override some or all aspects ofan implemented policy while maintaining execution of the policy.

FIG. 6 is a schematic drawing of a smart-home 200 having a localhousehold policy manager 202. In the current embodiment, the localhousehold policy manager 202 does not communicate over an externalnetwork (e.g., the Internet 205), as illustrated by the communicationsline 206. Instead, the household policy manager 202 may include a policylibrary containing a plurality of policies and/or an input/outputinterface, such as a universal serial bus port and/or storage cardinterface that enables stored policies to be provided to the householdpolicy manager 202. For privacy concerns, in some embodiments, it may bedesirable to block external communications from the household policymanager 202. This may alleviate data privacy concerns regardinghousehold data.

Further, as mentioned above, detected patterns may be useful inselecting policies for implementation within the home 200. The house 200is preferably equipped with pattern detection circuitry 204, which mayobserve patterns associated with the home 200. For example, the patterndetection circuitry 204 may mine data from smart devices 206 of the home200 to determine particular usage pattern of the devices 206 and/orhousehold activity patterns. The pattern detection circuitry 204 mayprovide discerned patterns to the household policy manager, which mayselect and/or suggest policies corresponding to the patterns forimplementation.

As will be discussed in more detail below, in one embodiment, thepattern detection circuitry 204 may detect that water is typically lefton when a member of the household brushes their teeth. This pattern maybe provided to the household policy manager 202, which may suggestand/or automatically implement a policy to provide notification tomembers of the household when they leave the water on when a member ofthe household brushes their teeth.

Capability-Based Policy Provision

As previously discussed in FIG. 5, some policy equipped homes 152 may beprovided/suggested policies based upon the capabilities of the smartdevices within the policy equipped home 152. FIG. 7 is a flow diagramillustrating a process 220 for implementing and/or suggesting policiesbased upon capabilities of smart-devices within the household, inaccordance with an embodiment. Further, FIG. 8 is a schematic drawing ofa household 250 capable of implementing the process 220 of FIG. 7. FIGS.7 and 8 will be discussed together.

The process 220 begins by determining smart device capabilities withinthe home 250 (block 222). The capabilities may include basicfunctionalities and/or sensor types (e.g., see, smell, hear, and/orimaging sensor, volatile organic compound sensor, audio sensor, and thelike) or the capabilities may be more complex (e.g., determine roomoccupancy from an infrared sensor). Further, the capabilities mayinclude a location where the functionalities may be performed and/orwhere the sensors are located.

In one embodiment, the household policy manager 154 is able to provide apoll request 252 to smart devices 254 within the home 250. The smartdevices 254 may provide a response 256 that defines the capabilities ofthe smart devices 254. For example, the smart device 254 “D1” may be asmart front-door lock. The smart device 254 “D1” may provide a response256 representing that the smart device 254 “D1” can provide occupancydetection and door locking at a front door 258. Further, smart device254 “D2” may be positioned in a living room and may include a speakerthat is capable of alarming. Accordingly, the response 256 from thesmart device 254 “D2” may represent that the smart device 254 “D2” canprovide an alarm in the living room.

In other embodiments, the capabilities may be discerned by obtaining aninventory of the available smart devices 254 within the home 250. Oncethe inventory of available smart devices 254 is complete, an alternativesource may be used to discern capabilities for the smart devices in theinventory based on their identities. For example, the household policymanager 154 may discern that smart device 254 “D1” and the smart device254 “D2” are within the home 250. In one embodiment, this discerning canbe carried out by having a user enter their UPC (Universal Product Code)information into a user interface. The household policy manager 154 maythen obtain a listing of capabilities of these devices from a database(e.g., local to or remote from the household policy manager 154) basedupon, for example, their UPC (Universal Product Code) information. Inother embodiments, more automated methods for discernment can be carriedout, such as by inquiry-response communications according to a standardprotocol, such as a Thread or Weave Protocol, by automated imagerecognition (e.g., a monitoring camera can visually “see” that there isa particular smoke detector or service robot within a room), and soforth.

Once the capabilities of the devices are determined, available policiesbased upon the capabilities may be determined (block 224). For example,the household policy manager 154 may select a subset of one or morepolicies that may use the smart device capabilities from a superset ofpolicies contained in a policy library.

In some embodiments, user-defined policies may be created using theavailable capabilities as inputs and/or outputs to the policy. Forexample, referring back to the home 250, a user-defined policy may becreated through an interface, where the interface provides the alarming,the occupancy detection, and/or the locking functionalities as availableinputs and/or outputs. Accordingly, an example of a user-defined policymight be: “When the occupancy detection functionality determines thatsomeone is at the front door, provide an indication in the living room,via the alarm functionality.”

Once the capability-based subset of policies is determined (e.g., asubset is selected from a policy library and/or user-defined policiesare created), these policies may be provided as a suggestion to a memberof the home 250 and/or may be automatically implemented (block 226). Forexample, as will be discussed in more detail below, the household policymanager 154 may begin monitoring data from the smart devices 254 and/ormay begin controlling the smart devices 254.

Demographic-Based Policy Provision

As discussed in FIG. 5, demographics may be used to provide householdpolicies. Demographics may include, for example, age, gender, education,marital status, socio-economic status, number in household, number inhousehold under age 18, language usually spoken at home, region ofresidence, race, ethnicity, religious affiliation, rural vs. urban, etc.FIG. 9 is a flow diagram illustrating a process 290 for obtainingpolicies based upon demographic information, in accordance with anembodiment. Sensor data that may be indicative of a householddemographic may be obtained (block 292). This sensor data may beanalyzed to infer demographic information about a household (block 294).By way of example, a video monitoring camera placed in the kitchen ofthe home can perform image processing on several days or weeks worth ofcaptured data to determine how many different individuals it sees on aregular basis, to establish how many occupants live in the house. Oncethe demographic information is known, the demographics may be used toselect a subset of policies typically applicable for the inferreddemographics.

FIG. 10 is a schematic drawing illustrating an example, in which a home310 is equipped with a household policy manager 154 that infersdemographic information to obtain relevant household policies, inaccordance with an embodiment. In the illustrated embodiment, sensordata (e.g., audio signatures 312, 314, and 316) are obtained from asensor (e.g., microphone 318. Demographic detection circuitry 320 mayinfer 322 that there is 1 adult male (e.g., based upon the audiosignature 316 being ascribed to a relatively low-pitch voice signature),1 adult female (e.g., based upon the audio signature 316 being ascribedto a relatively mid-range pitch voice signature), and 1 female child(e.g., based upon the audio signature 316 being ascribed to a relativelyhigh-pitch voice signature). The demographic detection circuitry 320 mayreside locally to the sensor (e.g., coupled to the microphone 318), mayreside between the sensor and the household policy manager 154, and/ormay reside at the household policy manager 154. Video data mayoptionally be used to confirm or to help arrive at such conclusions.

This demographic inference 322 may be used by the policy manager 154 toobtain policies 324 associated with the particular demographic 322. Forexample, in the current embodiment, the policy manager 154 may providethe demographic information 322 to an external policy service 324, whichmay return a subset of policies 324 associated with all or part of thedemographics 322. In alternative embodiments, the selection ofdemographic-applicable policies 324 may be local to the home 310.

In some embodiments, demographic information may be obtained withoutinferring demographics based upon sensor data. FIG. 11 is a schematicdrawing of a graphical user interface 350 for determining demographicinformation of a household, in accordance with an embodiment. Thegraphical user interface 350 may be provided, for example, on a smartdevice, such as a tablet computer, an alarm system display panel, etc.The graphical user interface may include any number of questions 352regarding demographic information of the household. In some scenarios,the graphical user interface 350 may auto-populate demographic itemsthat the demographic detection circuitry 320 has inferred, whileallowing the user of the graphical user interface 350 to modify thedemographic information.

Pattern-Based Policy Provision

Turning now to a discussion of pattern-based policy provision, FIG. 12is an schematic drawing illustrating an example of a pattern-basedpolicy implementation 370, in accordance with an embodiment. In theembodiment of FIG. 12, the household policy manager 154 (or otherpattern detection circuitry) may detect a pattern. For example, in thecurrent embodiment, text messages 372 and 374 may provide an indicationof an estimated home arrival time for a member of the household (e.g.,5:30 PM) on different days. The member of the household may arrive afterthe estimated home arrival time on both days (e.g., 5:50 PM on the firstday and 6:00 PM on the second day). Accordingly, the household policymanager 154 (or other pattern detection circuitry) may detect atardiness pattern. Accordingly, this pattern may be used by thehousehold policy manager 154 (or other policy provision service) tosuggest a policy 370 based upon this pattern. For example, in FIG. 12, apolicy 370 is suggested and/or automatically implemented to provide atext reminder to leave work at a time needed to get home by anysubsequently text-messaged estimated home arrival time. By suggestingthe policy 370 that provides a reminder, an inferred negative patternmay be averted in the future.

Household Awareness

Having discussed the manner in which household polices may reach thesmart-device environment 30, the discussion now turns to obtainingrelevant data from the smart devices 10 for implementing the policies.

As previously discussed, an inventory of smart devices and/or sensorsand their capabilities may be useful for constructing policies. FIG. 13is a schematic drawing illustrating a graphical user interface 400 forattributing data from the smart devices 10 to activities, events and/orattributes within the smart-device environment 30, in accordance with anembodiment. In the illustrated embodiment, the interface 400 includes anindication of the various smart devices and/or sensors in a house. Forexample, a basic home mock-up 402 may provide an illustration 404 ofrooms and placement 406 of smart devices and/or sensors. Further, thegraphical user interface 400 may provide an indication of capabilitiesof the smart devices and/or sensors. For example, the embodiment of FIG.13 includes capability clouds 408 indicating functionalities of smartdevices and/or sensor. In the illustrated example, the capability cloud408′ provides the capabilities of a ceiling mounted camera. Thecapabilities include audio playback (as illustrated by the speakericon), video monitoring (as illustrated by the eye icon), and audiomonitoring (as illustrated by the microphone icon). Capability cloud408″ illustrates a door lock smart device that can lock/unlock (asindicated by the pad lock icon) and provide occupancy detection at thelock (as indicated by the “OD” icon).

The capabilities of the smart devices 10 may be used to provideawareness within the smart-device environment 30. For example, audiosignals received at a microphone of one smart device 10 may beassociated with a particular household occupant. As mentioned in thediscussion of FIG. 10, a detected voice pitch (e.g., audio signature312) may be associated with a particular household occupant (orparticular type of household occupant (e.g., a male or female child)).In other examples, optical recognition (e.g., facial recognition orother image-based recognition), digital device presence (e.g., presenceof electronic devices associated with a particular person), or otherinputs may be associated with a particular household occupant orparticular type of household occupant.

In the example graphical user interface (GUI) 400 of FIG. 13, anawareness prompt 410 is generated. The awareness prompt 410 may prompt auser of the GUI 400 whether the system should attribute one or moreinputs 412 from the smart devices 10 as a particular activity, event,and/or attribute 414 of the household. For example, the prompt 410 isused to determine a relationship between an audio signature 412 and aparticular household member named Sydney. When the awareness prompt 410is confirmed (e.g., by a user selecting the “Yes” icon 416), anawareness rule 418 may be generated (e.g., in a database or data file420) to link the input 412 with the particular activity, even, and/orattribute 414 of the household. In some embodiments, when the awarenessprompt 410 is rejected (e.g., by a user selecting the “No” icon 420), anawareness rule 422 may be generated (e.g., in the database or data file420) to refuse a link between the input 412 with the particularactivity, even, and/or attribute 414 of the household.

In some embodiments, an awareness module 422 (e.g., hardware circuitryand/or software executed via a processor from tangible, non-transitorystorage) may generate awareness rules 422 without an awareness prompt410. For example, the awareness module 422 may receive inputs 412 (e.g.,from one or more smart devices 10) and automatically generate awarenessrules 418 without input from a user of the GUI 400. Automatic awarenessrules 418 may be useful when there is high confidence that the inputs412 should be associated with the particular activity, even, and/orattribute 414 of the household.

Once the awareness rules 412 are in place, reporting and/or controlrules 424 may be created. The reporting and/or control rules 424 (e.g.,household policies) may provide reporting when a certain particularactivity, event, and/or attribute 414 of the household occur. Further,these rules 424 may trigger events based upon the particular activity,event, and/or attribute 414. For example, in the illustrated embodiment,a front door rule specifies that a door is to remain locked when Sydneyis home alone. Thus, awareness of Sydney's lone occupancy in thehousehold triggers persistent locking at the front door. Any number ofevents can be triggered by the rules 424. Rules 424 may triggerfunctionalities of smart devices (e.g., smart devices 426),processor-based functionalities (e.g., sending an email, posting acomment on social media, setting a calendar entry, providing a cellularphone notification, etc.), and so forth.

As may be appreciated, reporting and/or control based upon householdpolicies may rely on sensor confidence regarding household awareness.FIGS. 14-16 illustrate processes for providing sensor confidence and/orsensor confidence improvement, in accordance with certain embodiments.FIG. 14 is a flow diagram illustrating a process 480 for providingsensor confidence related to a particular policy, in accordance with anembodiment. First, a level of sensor awareness confidence is determinedfor existing policies (e.g., the rules 424 to be implemented within thehousehold) (block 482). Sensor awareness confidence is a level ofconfidence that particular sensor inputs 412 may accurately beassociated with the particular activity, event, and/or attribute 414 forthe policies to be implemented. For example, in the door locking rule424 of FIG. 13, sensor awareness confidence may relate to a) detectingSydney's presence within the home and/or b) whether there is only oneperson (e.g., Sydney) currently occupying the home. For someembodiments, each sensor awareness confidence level can be expressed asa single scalar percentage metric, while for other embodiments it can beexpressed as a multiple-feature vector whose components can be processedto arrive at the single scalar percentage metric, or that can beprocessed individually or in sub-groups to arrive at variouscomponent-level confidence determinations.

Once the confidence level is determined, a determination is made as towhether the confidence level meets a confidence threshold (decisionblock 484). For example, confidence may be increased by observingadditional inputs 412 that may be attributed to the particular activity,event, and/or attribute 414 for the policies to be implemented. If theenvironment has not experienced enough input 412 data to provide athreshold level of subsequent awareness (e.g., the confidence level hasnot met a confidence threshold), monitoring may be continued andadjustments to input 412 ranges associated with the particular activity,event, and/or attribute 414 for the policies to be implemented may occur(block 486).

Once an awareness confidence level meets a threshold confidence level,the existing policies using the awareness may be implemented (block488). For example, in the door locking example of FIG. 13, the door maybe locked when the smart environment is sufficiently confident thatSydney is home alone.

In addition to implementing policies upon a particular sensor confidencethreshold, it may be beneficial to predict sensor confidence prior toimplementation of a particular household policy. FIG. 15 is a flowdiagram illustrating a process 500 for presenting a predicted sensorconfidence for a new policy, in accordance with an embodiment. First, aparticular household policy or set of policies is selected (block 502).Next, a determination is made as to the sensor attributes needed todiscern the conditions (e.g., inputs sourced from smart devices 10)(block 504). For example, in the door locking example of FIG. 13, sensorattributes relating to 1) identifying Sydney as an occupant and 2)identifying that an occupant is home alone may be needed. Accordingly,video data, audio data, infrared data, etc. useful for identifying anoccupant as Sydney and/or a quantity of occupants may be obtained in thedetermination step of block 504. In some embodiments, the needed sensorattributes may be discerned by forming an aggregation of awareness rules418 for a particular household policy.

Once the sensor attributes are determined, a determination is made as tothe level of precision with which the determined sensor attributes maybe observed (block 506. For example, low quality audio and/or videosensors may be less precise in observing audio and/or video attributes,especially in noisy and/or low-light environments. Further, infraredsensor precision may be affected by environmental variables.Accordingly, to determine precision of sensor attribute observation,sample data may be obtained from the smart devices 10, which may beanalyzed for data quality (e.g., precision of detail in the data).

Based upon the determined precision, an indication of predicted sensorconfidence for obtaining the sensor attributes for the policy may beprovided (block 508). For example, the predicted sensor confidenceindication may include a indication that a living room camera may have a50% chance of correctly identifying Sydney as the occupant.

The predicted sensor confidence may be useful in determining potentialmodifications to a sensor arrangement and/or environmental arrangement,such that increased sensor confidence may be obtained. For example,sensor variables such as placement, number of sensors, etc. may bemodified to increase sensor confidence. Further, environmental variablesuch as, lighting, noise levels, etc. may be altered to increase sensorconfidence. FIG. 16 is a flow diagram illustrating a process 520 forsuggesting modifications to enhance sensor confidence, in accordancewith an embodiment.

The process 520 begins by determining the predicted sensor confidence(e.g., via the process 500 of FIG. 15) (block 522). Next, variables thatwould increase the sensor confidence are determined (block 524). Forexample, in some embodiments, a number of variables associated withparticular devices may be provided that alter the precision of thesensors. For example, noise attributes for audio sensors and/or lightfor video sensors. In some embodiments, environmental variables, such asenvironment temperature, room size, etc. that may alter sensor precisionmay also be provided.

Next, variable modifications may be suggested to obtain better attributeobservation (block 526). For example, in audio sensors, interference(e.g., audio noise) may result in less sensor precision. Based uponsamples obtained during the process 500, it may be determined that thereare noisy components (e.g., an overhead fan) near the audio sensor.Accordingly, replacement of the audio sensor to a less noisy area may besuggested.

The modification suggestions may be made in a number of ways. In someembodiments, a user interface (e.g., a graphical user interface, anaudio interface, etc.) may provide the suggestions. Further, thesuggestions may be provided via one or more of the smart devices 10 orother electronic devices associated with the household. FIG. 17 is aschematic drawing illustrating examples of suggestions 540, 542, 544,and 546 that may be result of the process of FIG. 16, in accordance withan embodiment. The suggestions 540, 542, 544, and 546 may be acommunicative message provided to one or more household members. Forexample, the suggestions 540, 542, 544, and/or 546 may be a textmessage, pop-up notification, or audio track provided via a householdmember's smart phone or other electronic device.

As mentioned above, the suggestions may include sensor and/orenvironmental variable modifications. For example, suggestions 540, 542,and 544 each illustrate sensor modifications, while suggestion 546illustrates an environmental modification suggestion. Suggestion 540provides a suggestion to add additional sensors for increased accuracy.Suggestion 542 suggests that a sensor be re-positioned in an alternativelocation (e.g., from bathroom 1 to bedroom 3). Suggestion 544 suggeststhat a sensor be re-oriented, providing an orientation illustration 548.

In contrast, the environmental modification suggestion 546 suggestsenvironmental changes. In the example suggestion 546, an ambient lightmodification may increase the awareness accuracy of the sensors (whichmay, for example, increase the ability of a video monitoring device toyield images or image sequences with better dynamic range from whichfaces may be better recognized). Other environmental changes mightinclude a temperature adjustment, noise adjustment, etc.

Additionally, the suggestions may provide an indication of approximateimprovement (e.g., 72% improvement in suggestion 540 and/or confidenceimprovement from 63% to 98% in suggestion 542) or may provide a generalindication of improvement (e.g., suggestions 544 and 546). In someembodiments, the suggestion may provide a modification without providingan indication of improvement.

When modifications are made to the smart home environment (e.g., basedupon the suggestions of FIG. 17, adjustments may be needed to properlyassociate the sensor attributes with a policy condition. FIG. 18 is aflow diagram illustrating a process 560 making adjustments based uponmanual modifications of a policy, in accordance with an embodiment.First, a policy may be implemented according to a set of associationrules and controls, as described above with regards to FIG. 13 (block562). A determination is made as to whether modifications are detectedwithin the environment (decision block 564). For example, the system maydetermine that the orientation of a sensor, the location of a sensor,etc. has changed.

When no changes are detected, the policy implementation continues (block562). However, when a modification is detected, a determination is madeas to whether the modification warrants a change to the policy (e.g., tothe association rules used for reporting and/or control in the policy)(decision block 566). In one example, when an audio sensor is moved froma bedroom to a bathroom, additional audio interference (e.g., from arunning sink) may be present. Accordingly, the association rules may beupdated to filter out audio signals from a running sink when associatingsensor data with a particular policy condition.

If no modifications to the policy are needed, the policy implementationcontinues (block 562). However, when a modification is needed, anadjustment may be made to the policy (block 568). For example, in theaudio sensor example, an audio filtering may be added to associationrules for the audio sensor. In the case of additional sensors beingadded to the system, new association rules may be added to account fordata obtained via the new sensors. Once the adjustments are made, thepolicy is implemented with the new adjustments (block 562).

Context-Based Awareness Embodiments

Turning now to a more detailed discussion of particular awarenessprocesses, FIG. 19 is a flow diagram illustrating a process 590 formonitoring and becoming aware of activities of a household, inaccordance with an embodiment. First, a sensor input that is indicativeof a particular activity is detected (block 592). For example, aninfrared input, a video input, and/or an audio input may indicateoccupancy within a room of a household. Next, characteristics of theactivity are inferred using characteristics of the input (block 594).These inferred characteristics may provide an indication whether or nota household policy condition (or a portion of a household policycondition) is met. For example, an infrared signature, audio signature,and/or video signature may be used to identify a particular householdmember occupying a room. The inferred characteristics may be recordedand/or reported (block 596) (e.g., to the household policy manager,which may provide reporting and/or control within the household system).

Particular inferences may be made based upon the context of the sensors(e.g., the sensor placement within the house or other contextualinformation regarding the sensors). FIG. 20 is a schematic drawingillustrating policies implemented according to locational zones, inaccordance with an embodiment. In one example, the smart-deviceenvironment 30 may be broken up into localized zones, such as a diningroom zone 610, a living room zone 612, office zone 614, kitchen zone616, bathroom zone 618, and bedroom zones 620 and 620′. Further, zonesmay encapsulate multiple rooms and/or areas. For example, in theembodiment of FIG. 20, a whole-house zone 622 is present. Further, anupstairs zone 624 and a zone 626 including all bedrooms is included.

The various zones may be used differently to provide awareness to thehousehold policy management system. For example, awareness of aparticular occupant's presence may be discerned using similarassociation rules when the zones are similar. However, when the zonesdiffer (e.g., different ambient light and/or noise), the associationrules may vary from zone to zone.

Further, a context of a particular zone may provide additional insightinto the activities, events and/or characteristics to be inferred. Forexample, when an occupant is in the dining room zone 610, it may be morelikely that the occupant is eating dinner than performing otheractivities (e.g., watching television). Similarly, an occupant in abathroom is more likely to be brushing their teeth than performing otheractivities (e.g., eating dinner). Accordingly, a zonal context may beused in the inference of activities, events, and/or characteristics ofthe house.

Turning now to a more focused discussion of zonal inferences, FIG. 21 isa schematic drawing of a household policy system 640 having a diningzone 610, in accordance with an embodiment. FIG. 22 is a flow diagramillustrating a process 650 for monitoring dining, in accordance with anembodiment. Because they are related, FIGS. 21 and 22 will be discussedtogether.

As illustrated the dining zone 610 may have one or more sensors (e.g.,smart devices 10) that may accumulate data from the dining room zone610. For example, the smart devices 10 in the zone 610 may include videosensors, audio sensors, infrared sensors, vapor detection sensors (e.g.,VOC and/or other compound detection and identification sensors), etc.

In some embodiments, it may be beneficial to become aware of householdeating patterns (e.g., whether the household eats at scheduled times,whether the household eats in the dining room, whether the householdeats together, etc.). Using the smart devices 10, the system 640 maymonitor the dining room for activity (e.g., changes to the smart devicesensor inputs) (block 652). Upon detecting such activities, a context ofthe activity is inferred (block 654). For example, an audio signaturematching a dining chair 642 movement across a floor 644 may suggest thatan occupant is sitting in the chair 642 (e.g., because the occupant mayhave presumably moved the chair 642 to sit in it). Indeed, video inputsmay confirm and/or identify that occupants are sitting in the chair 642and/or at the table 646. Additionally, smart device 10 inputs may beused to obtain a number of contextual clues, such as utensil movement,conversation content, vapor detection, etc. For example, in oneembodiment, the vapor sensors may detect the presence of food within thedining room zone, which may indicate that a meal is being consumed inthe dining room zone 610.

Based upon the determined context, particular awareness questions may bedetermined. For example, in the process 650, a determination is made asto whether the context suggests that one or more household occupants iseating a meal (decision block 656). Any number of decision blocks 656may exist, depending on particular inferences that are to be made foruse with the household policies. For example, an additional decisionblock 656′ may determine whether the context suggests that an occupantis working at the dining room table, etc.

When the decision block (e.g., decision block 656) is answered in theaffirmative, an inference is made, ascribing the inputs to the activityof the decision block. For example, in the process 650 when the contextsuggests that a meal is being eaten in the zone 610, the activity in thedining room is ascribed to eating a meal (block 658) and is reportedand/or stored for use by the household policy manager (block 660). Aswill be discussed in more detail below, the reported and/or storedfindings may be useful for reporting and/or control activities sourcedfrom the system (e.g., the household policy manager 154).

Turning now to the living room context, FIG. 23 is a schematic drawingof a system 670 for monitoring activities in a living room zone 612, inaccordance with an embodiment. As illustrated in FIG. 23, the livingroom zone 612 may have one or more sensors (e.g., smart devices 10) thatmay accumulate data from the zone 612. For example, the smart devices 10in the zone 612 may include video sensors, audio sensors, infraredsensors, vapor detection sensors, etc. In addition, a television 672 maybe a smart device 10, capable of providing data relating to thetelevision 672 and/or the living room zone 612.

The sensors may be used to become aware of activities, events, and/orcharacteristics of the living room zone 612. For example, FIGS. 24 and25 are flow diagrams illustrating processes for monitoring televisionuse, in accordance with an embodiment. In FIG. 24, a process 700 fortelevision usage awareness is provided. When a determination is makethat individuals are in a room with a television (e.g., in the livingroom zone 612) (block 702), a determination is made as to whether thetelevision 672 is on (decision block 704). Block 702 may discernoccupancy via any number of occupancy detection mechanisms, such asinfrared changes attributable to an occupant, audio signaturesrepresentative of an occupant, video signatures representative of anoccupant, etc. Further, a particular identity of an occupant may bediscerned in block 702. For example, an audio and/or video signaturerepresentative of a particular occupant may be used to identify theoccupant.

Further, the determination regarding whether the television 672 is onmay be discerned based upon data provided by the television 672 and/ordata obtained by the other devices 10 (e.g., video detection of anactive television 672 screen and/or audio from the television 672speakers). If the television 672 is not on, monitoring continues untilthere are changes in the television 672 status. When the television 672is on, the system may determine whether there is a prescribed television672 allotment for the household and/or a particular household memberoccupying the zone 612 (decision block 706). If there is no prescribedallotment, processing continues, as will be discussed below.

When there is a prescribed allotment, the system may report and/orrecord the television 672 usage as part of the prescribed allotment(e.g., the household policy manager 154 (block 708). Accordingly, thesystem (e.g., the household policy manager 154) may subsequently controlthe television 672 based upon the allotment and usage, as will bedescribed in more detail below.

In conjunction with (or independently with) the allotment and usageawareness, the processing may include a determination of whether or notactivities (e.g., watching television) are being performed alone or as agroup (decision block 710). For example, if multiple occupants aredetected, the activity may be attributed to an activity that isperformed with other household members (block 712). When there is asingle occupant, the activity may be ascribed to an activity that isperformed alone (block 714). In either case, the ascribed findings maybe reported and/or recorded (e.g., to the household policy manager 154)(block 716).

It may be beneficial to become aware of additional information whennon-parental supervision is present. Accordingly, awareness mayheightened when non-parental supervision is present. FIG. 25 illustratesanother living room awareness process 730 providing such heightenedawareness.

Similar to process 700, the system may determine who is in the livingroom zone 612 (block 732). Further, the system may discern whether thetelevision 672 is on (decision block 734).

The process 730 may include a determination (block 736) of whetherchildren are being supervised by parents or non-parents (e.g., ababysitter or grandparents). If the children are being supervised byparents, monitoring continues until non-parental supervision occurs.

When non-parental supervision is present, alternative awareness mayoccur. For example, in the illustrated embodiment, heightened awarenessof television usage is inferred (block 768). For example, the heightenedtelevision 672 awareness may include awareness regarding: content beingwatched, an amount of time of television 672 usage during thenon-parental supervision, who is participating in the television 672usage, etc. This awareness may be reported and/or recorded to the system(e.g., for subsequent reporting and/or control by the system) (block740).

Turning now to an office context, FIG. 26 is a schematic drawing of asystem 770 for monitoring office zone 614 activities, in accordance withan embodiment. The zone 614 may have one or more sensors (e.g., smartdevices 10) that may accumulate data from the office zone 614. Forexample, the smart devices 10 in the zone 614 may include video sensors,audio sensors, infrared sensors, vapor detection sensors, etc.Additionally, the zone 614 may include one or more computers 772 and/orhandheld electronic devices 774. The zone 614 may include a desk 776and/or a chair 778.

Awareness of activities, events, and/or characteristics of the zone 614may be useful in household policies. FIG. 27 is a flow diagramillustrating a process 780 for monitoring office activities, inaccordance with an embodiment. One activity of interest may be internetusage awareness, which may be obtained via blocks 782. Another activityof interest may be awareness regarding working, which may be inferredusing blocks 784.

Discussing first the internet usage awareness, internet activity may bedetected (block 786). This may be done by monitoring network packets, byobserving computer 772 and/or electronic device 774 communications, etc.

An individual causing the internet activity is then determined (block788). For example, if the computer 772 and/or electronic device 774 isdetermined to be the device causing internet activity, a householdmember operating the computer 772 and/or electronic device 774 may bediscerned. In some embodiments, this may be done by determining acurrently executed login credential for the computer 772 and/orelectronic device 774, determining a user typically associated with thecomputer 772 and/or electronic device 774, and or by using a smartdevice 10 to identify an individual interacting with the computer 772.Findings regarding the internet usage may be reported and/or recordedfor subsequent control and/or reporting (block 790).

Turning now to awareness regarding working, an analysis of sensor datamay be used to determine if a household member is working (block 792).For example, an audio signature of keyboard clicking, a desk chairmoving, and/or papers shuffling etc. may indicate that someone isworking.

Next, a particular household member may be identified as the persondoing the work (block 794). For example, the member may be identifiedbased upon image recognition (e.g., a video signature), audiorecognition (e.g., an audio signature), etc. Any findings may berecorded and/or reported for subsequent control and/or reporting by thesystem (block 790).

Turning now to a kitchen context, FIG. 28 is a schematic drawing of asystem 800 for monitoring activities in a kitchen zone 616, inaccordance with an embodiment. The kitchen zone 616 may include one ormore sensors (e.g., smart devices 10) that may accumulate data from thekitchen zone 616. For example, the smart devices 10 in the zone 616 mayinclude video sensors, audio sensors, infrared sensors, vapor detectionsensors, etc. Additionally, the zone 616 may include a refrigerator 802,a pantry, 804, and kitchen appliances 806, which may be monitored by thesmart devices 10. In some embodiments, the refrigerator 802 and/or theappliances 806 may be smart devices 10 capable of providing data to thesystem 800.

FIG. 29 is a flow diagram illustrating a process 810 for monitoringactivities in the kitchen zone 616, in accordance with an embodiment.One or more attributes of the refrigerator may be determined (block812), the term “attributes” including activities or events associatedwith the refrigerator in addition to features or characteristics. Forexample, attributes such as when the refrigerator door is opened and/orclosed, a duration that a door is opened, an amount of water dispensedthrough a water dispenser, a level of ice in the ice maker, etc. may bedetermined.

Additionally or alternatively, attributes of the pantry 804 may bedetermined (block 814). For example, attributes such as when the pantry804 is opened/closed, duration that pantry 804 is open, items removedand/or added to the pantry 804, etc. may be obtained.

Further, attributes of the appliances 806 may be determined (block 816).For example, attributes such as status of the appliances 806 (e.g., onor off), duration of utilization of the appliances 806, etc. may beobtained.

Using the determined attributes of the refrigerator 802, the pantry 804,and/or the other appliances 806, additional information may bediscerned. Any findings of the zone 616 may be recorded and/or reportedto the system (e.g., for subsequent reporting and/or control by ahousehold policy manager 154.

Turning now to the bathroom zone 618 context, FIG. 30 is a schematicdrawing of a system 830 for monitoring activities of the bathroom zone618, in accordance with an embodiment. FIGS. 31-33 are flow diagramillustrating processes for monitoring bathroom activities whilepreserving personal privacy, in accordance with embodiments. Thebathroom zone 618 may include one or more sensors (e.g., smart devices10) that may accumulate data from the bathroom zone 618. For example,the smart devices 10 in the zone 618 may include video sensors, audiosensors, infrared sensors, vapor detection sensors, etc. Additionally,the zone 618 may include lighting 832, one or more sinks 834, a shower836, a bathtub 838, and/or a toilet 840. In some embodiments, thelighting 832, the one or more sinks 834, the shower 836, the bathtub838, and/or a toilet 840 may be smart devices 10.

One activity, event, or characteristic that may be interesting regardinghousehold policies may be water usage. FIG. 31 provides a process 850for determining water usage attributes for a bathroom zone 618. Theprocess 850 begins by detecting running water (block 852). For example,this may be determined based upon vibration analysis (e.g. of pipes),audio signatures, and/or video signatures within the bathroom zone 618.

The running water may be attributed to a particular water-basedapplication (e.g. a sink 834, a shower 863, a bathtub 838, and/or atoilet 840) (block 854). For example, this may be accomplished basedupon a determination of an approximate location of the running water,etc. Any findings regarding the water usage may be reported and/orrecorded within the system (e.g., for subsequent control and/orreporting by the household policy manager 154) (block 856).

Another bathroom zone 618 characteristic of interest may be light usage.FIG. 32 provides a process 860 for determining light usage within azone. First, light usage is detected (block 862). In the case where thelighting 832 is a smart device 10, the lighting 832 may provide lightusage data to the system. Additionally and/or alternatively, light usagemay be detected via infrared sensors and/or video (e.g., imagingsensors).

Characteristics regarding the light usage may also be determined (block864). For example, the time of day, occupancy information (e.g., who isoccupying the zone), ambient lighting conditions, lighting leveladjustments, lighting duration, etc. may be determined. These findingsmay me reported and/or recorded within the system (e.g., for subsequentcontrol and/or reporting by the household policy manager 154) (block866).

FIG. 33 illustrates an example of process 880 for awareness regarding abathroom zone 618 activity. Specifically, the embodiment of FIG. 33begins by detecting that an occupant of the zone 618 is brushing theirteeth (block 882). For example, an audio signature and/or videosignature may be associated with the sounds and/or images of teethbrushing in the zone 618. Next, additional characteristics may bedetermined (e.g., the sink 834 being left on, a duration of teethbrushing, a speed of teeth brushing, etc.) (block 884). These findingsmay me reported and/or recorded within the system (e.g., for subsequentcontrol and/or reporting by the household policy manager 154) (block886).

Reported and/or recorded activities/findings may be compiled, such thatcomplex findings may be obtained. FIG. 34 is a flow diagram illustratinga process 900 for reporting compiled activities, in accordance with anembodiment. The process 900 begins my compiling the reported and/orrecorded findings provided to the system (e.g., each finding from everyroom and/or device in the house) (block 902). The compiled findings maybe filtered based upon common activities, events, and/orcharacteristics. For example, findings may be filtered based upon aparticular household member being involved in the activity (e.g., all ofAllie's activities), based upon household member activities (e.g.,brushing teeth), and/or events (e.g., active Internet usage and/oractivated lights).

The compiled findings may be used to determine individual activityallocations (e.g., all of Allie's activities or all of Allie's teethbrushings) (block 904). Further, the compiled findings may be used todetermine household activity allocations (e.g., household utilityconsumption, etc.) (block 906). The individual activity allocationsand/or household activity allocations may be reported and/or recorded(block 908) for use by the system. For example, FIG. 35 is a schematicdrawing of a graphical user interface 910 useful for reporting compiledactivities, in accordance with an embodiment. In the interface 910, ahousehold member's (e.g., Allie's) activities are displayed by the piechart 912. Further, the pie chart portion 914 provides furthercharacteristics and/or details relating to the portion 916 of the chart912. In the illustrated embodiment, details relating to Allie's internetusage are more particularly detailed in the portion 914.

Inferred Activities

In some situations, observed characteristics, behaviors, and/oractivities may be used to infer other characteristics, behaviors, and/oractivities. For example, some monitored activities may not be directlyobservable, but may be monitored indirectly via inferences made by thehousehold policy manager.

For example, in one embodiment, mischief may be inferred based uponobservable activities of individuals in a household. FIG. 36 is a flowdiagram illustrating a process 930 for detecting child mischief, inaccordance with an embodiment. Specifically, the process 930 begins bydetecting child occupancy or occupancy of other classes of occupants tobe monitored (block 932). In some embodiments, these classes ofoccupants may be determined based upon audio monitoring, optical (e.g.,video) monitoring, or other monitoring (e.g., infrared occupancymonitoring). For example, characteristics of audio signatures, such asspeech patterns, pitch, etc. may be used to discern child occupancy.Next, the occupants may be monitored, specifically listening forlow-level audio signatures (e.g., whispering or silence), while theoccupants are active (e.g., moving or performing other actions) (block934). Based upon the detection of these low-level audio signaturescombined with active monitored occupants, the system may infer thatmischief (e.g., activities that should not be occurring) is occurring(block 936).

In some embodiments, some particular activities or other context may beused to infer that mischief is occurring, or to exclude an inferencethat mischief is occurring. For example, it may be expected that certainactivities be performed in quiet, thus indicating that the quietactivity is unlikely to be mischief. For example, reading a book,mediating, etc. are oftentimes performed in quiet. Additionally,contextual information such as occupancy location may be used to excludean inference of mischief. For example, if the occupancy occurs in alibrary, study, or other area where quiet activity may be expected,inference of mischief may be excluded. Accordingly, when theseactivities are performed, the system may infer that mischief is notoccurring, despite low-level audio signatures in parallel with activeoccupants.

Additionally or alternatively, in some embodiments, particularactivities or other contextual information may be used to provide astronger inference of mischief. For example, when children are near aliquor cabinet or are in their parents' bedroom alone, the system mayinfer that mischief is likely to be occurring.

Upon inferring that mischief is occurring, the system may report and/orrecord the findings for subsequent use (block 938). For example, thefindings may be used in household reporting, near real timenotification, or may be used to control smart devices within the home(e.g., provide a verbal warning regarding the inferred mischief via aspeaker of a smart device (e.g., a smoke detector). Accordingly, thefinding may lead to deterrence of further mischief within the home.

It may be beneficial to monitor the emotional state of occupants withina household. FIG. 37 is a flow diagram illustrating a process 960 forinferring an emotional state of occupants, in accordance with anembodiment. Certain contextual queues of individual occupants may beused to infer the emotional state of household occupants. For example,crying may signify a sad emotional state, whereas laughing may signify ahappy emotional state.

Further, some general cultural norms, which can often be keyed tohousehold location, may be used in universally inferring an emotionalstate of household occupants. For example, a head nod up and down in ahousehold located in a first country of the world may signify anagreeable state, whereas a head nod side to side may signify the sameagreeable state in a household located in a second country of the world.

As mentioned above, emotional state may be more accurately inferred byunderstanding a context of the occupant. Accordingly, the process 960may optionally begin by discerning an identity or class (e.g., sex, age,nationality, etc.) of the occupant (block 962).

The process 960 also includes detecting emotional context data (block964). For example, optical indicators of facial expressions, headmovement, or other activities of occupants may be used to infer anemotional state. Additionally, audio queues, such as audio signatures ofcrying, laughing, elevated voices, etc. may be used to infer emotions.Further, infrared information, such as body temperature, etc. may beused in an emotional state inference.

Based upon the obtained emotional context data, an inference may be madeas to the occupant's emotional state. As mentioned above, the inferencemay be generalized (e.g., based upon context clues for an entire classof occupants) or may be particularly tailored for a particular occupant.For example, a generalized inference of a happy emotional state of theoccupant may be made when a visual indication of laughter and/or anaudio indication of laughter is obtained. In contrast, particularlytailored inferences may look at the emotional context data in view of aparticular occupant's known characteristics. For example, the system mayknow that Sue cries both when she is happy and when she is sad.Accordingly, the system may discern that an inference of Sue's emotionalstate based upon crying alone would be weak. However, the system mayalso know that Sue typically smiles when she is happy and maintains astraight face when she is sad. Accordingly, when a visual indicationshows that Sue is crying and has a straight face, the system may inferthat Sue is sad. This inference may be different for other occupants,because the particularly tailored example uses particular emotionalindicators of Sue.

Upon inferring the emotional state of the occupant, the system mayreport and/or record the findings for subsequent use (block 968). Forexample, the findings may be used in household reporting, near real timenotification, or may be used to control smart devices within the home.

In some embodiments, it may be beneficial to monitor bullying. FIG. 38is a flow diagram illustrating a process 980 for detecting bullyingand/or other undesirable actions, in accordance with an embodiment. Theprocess 980 begins with monitoring contextual data (block 982). Forexample, audio monitoring, optical monitoring, infrared monitoring, etc.may be used to discern occupancy and undesirable activities of theoccupants. In one embodiment, the contextual data may include audiosignatures indicating “bully” keywords such as derogatory name-calling,elevated voices, etc. Accordingly, the system may monitor for and detectthe use of such “bully” keywords (block 984). Additionally, in someembodiments, the contextual data may include audio signatures indicatingthe use of foul language. Accordingly, the system may monitor for anddetect the use of such foul language by the occupants (block 986). Insome embodiments, the contextual data may include the identity of theindividuals interacting with one another. Accordingly, the system maydetect the identities of the individuals occupying the space where theundesirable activities may be occurring (block 988). Based upon thecontextual data, an inference may be made that bullying and/or otherundesirable activities are occurring (block 990). The findings of thisinference may be reported and/or recorded for subsequent use (block992).

In some embodiments, it may be desirable to detect interaction withundesirable substances. For example, it may be desirable to understandwhen occupants are interacting with undesirable substances. FIG. 39 is aflow diagram illustrating a process 210 for inferring interaction withundesirable substances, in accordance with an embodiment. Process 210begins by detecting occupancy in an environment (block 212).Additionally, the process 210 includes monitoring for contextual datarelated to interaction with undesirable substances (block 214). Forexample, a chemical detection system/volatile organic compound sensormay be used to detect the presence of undesirable activities (e.g.,poisonous compounds, alcohol, tobacco, etc.). Additionally, thecontextual data may include: occupancy in proximity to locations wheresuch undesirable substances are stored (e.g., occupancy near a liquorcabinet, near a kitchen cabinet containing poisonous kitchen cleaningproducts, etc.); visual indication of interaction between the occupantand the undesirable substance; etc. Based upon the contextual data, thesystem may infer interaction of an occupant with the undesirablesubstances (block 216). Accordingly, the inference may be reportedand/or recorded for subsequent reporting and/or control (block 218).

Additionally, the system may be used to monitor chore completion. FIG.40 is a flow diagram illustrating a process 240 for detecting chorecompletion status, in accordance with an embodiment. The process 240begins with monitoring contextual data associated with daily chores(block 242). For example, on trash pickup days, the system may monitorfor data such as audio and/or video associated with the removal of trashbags from a trashcan. Another example may include monitoring audioand/or video of dusting activities, vacuuming activities, moppingactivities, etc. Further, the contextual data may include audio and/orvideo data indicating study/homework activities, instrument practice,etc. Based upon the contextual data, inferences may be made regardingwhether chores have been completed (block 244). The inferences may bereported and/or recorded for subsequent reporting and/or control (block246).

FIG. 41 is a flow diagram illustrating a process 250 for monitoringmedical symptoms, in accordance with an embodiment. First, contextualdata associated with medical conditions may be monitored (block 252).For example, certain movement patterns have been associated withAlzheimer's disease. In some embodiments, the system may monitorhousehold occupants' movement patterns and compare these movementpatterns with those associated with Alzheimer's disease. In otherembodiments, the system may track audible and/or visual cues associatedwith maladies, such as: coughing, sneezing, aching, etc.

Next, based upon the monitored contextual data, an inference may be maderegarding medical diagnosis of one or more occupants of the household.For example, when an occupant's movement patterns match those of themovement patterns associated with Alzheimer's disease, the system mayinfer a higher probability that the household occupant has the disease.Further, upon attributing coughs and/or sniffing, for example, by ahousehold occupant, the system may infer that the occupant is acquiringa cold and/or influenza. The inference may be reported and/or recordedfor subsequent use in the system.

Situational Observation

In some instances, when particular situations arise, additionalmonitoring/inferences may be desired. FIG. 42 is a flow diagramillustrating a process 260 for situational monitoring, in accordancewith an embodiment. The process 260 begins by discerning if a situationwarranting additional monitoring/inferences is occurring (decision block262). If no such occurrence is detected, routine monitoring proceeds(block 264). However, when a situation warranting additionalmonitoring/inferences is detected, contextual data associated with thesituational monitoring/inference policies may be monitored (block 266).The system may infer activities based upon the contextual data (block268). These inferences may be reported and/or recorded for subsequentreporting and/or control (block 270).

In one embodiment, additional monitoring may occur when a child is homealone. Any number of additional monitoring tasks and/or inferences maybe made when children are home alone. For example, the system might makemore sensitive inferences when children are home alone, may detect thepresence of unexpected occupants, etc.

FIG. 43 is a flow diagram illustrating a process 290 for situationalmonitoring of children who are home alone, in accordance with anembodiment. First, the system may detect that a child is home alone(block 292). For example, the occupancy sensors may discern occupancyand identification data may be used to discern that the occupants areall children. Accordingly, the system may discern that children are homealone.

In the embodiment of FIG. 43, when children are home alone, a policy isset to inform parents when occupants approach and/or enter the house.Accordingly, the system may detect, via occupancy sensors, occupants atthe front door (block 294). Upon such detection, the system may notifyparents of the approaching occupant (block 296).

Another situation that may warrant additional monitoring may besupervision of children by third parties (e.g., a babysitter orgrandparents). FIG. 44 is a flow diagram illustrating a process 310 forsituational monitoring of non-parental supervision, in accordance withan embodiment. First, the system determines whether a third party (e.g.,a babysitter) is providing supervision (decision block 312). Forexample, the system may discern that parents are occupying the housewith the children, thus indicating parental supervision. Alternatively,the system may discern that only third parties occupy the house with thechildren, thus indicating third party supervision.

If there is no third party supervision, routine monitoring proceeds(block 314). However, when third party supervision is detected,additional monitoring may occur. Any number of additional monitoringtasks may be desired when third party supervision is detected. Forexample, the sensitivity of cautionary inferences (e.g., interactionwith inappropriate content and/or substances) may be heightened, theactions of the third parties can be modified, additional actions of thechildren, and etc. may be monitored.

In the embodiment of FIG. 44, a situational policy is set to monitorinteraction between the third party and the children (block 316),monitor the electronic device (e.g., television, tablet, or computer)usage amount and/or usage characteristics (e.g., consumed content) ofthe children (block 318), and monitor for unexpected occupantsapproaching and/or entering the house (block 320). Findings regardingthese additional monitoring tasks are reported and/or recorded forsubsequent reporting and/or control (block 322).

Another situation that may warrant additional monitoring/inferences is asituation where assertion/promises are made. FIGS. 45 and 46 are flowdiagrams illustrating processes 350 and 390 for monitoring assertionactivities, in accordance with embodiments. Process 350 is a generalprocess for assertion-based monitoring and process 390 is a morespecific assertion based policy regarding an appointment assertion.Starting first with process 350 of FIG. 45, the process 350 begins bydetecting an assertion/promise (block 352). In some embodiments, thesystem discerns such a promise by monitoring voice conversations withinthe household and/or on a portable electronic device (e.g., a tablet,computer, or cellular telephone). In some embodiments, suchassertions/promises may be detected by monitoring text messaging (e.g.,SMS text messages, etc.). In some embodiments, assertions may beobtained from email, calendar, or other electronic device applications.

Regarding the appointment-based assertion monitoring process 390, thesystem may detect an appointment assertion (block 392). In one example,a text message may be provided by a household member stating, “I'll behome by 5:00.” Accordingly, an assertion may be detected by monitoringthe text messaging of an electronic device. Alternatively, anappointment may exist in an electronic calendaring application of anelectronic device, which may be used to discern an appointment.

Returning to process 350, a determination is made as to whether theassertion/promise is capable of being completed (decision block 354 ofFIG. 45 and decision block 396 of FIG. 46). For example, returning tothe “I'll be home by 5:00” assertion, the system may determine a currentlocation of the household member and how long it would take to get fromthe current location to the occupant's house (block 394 of FIG. 46). Ifthe occupant's house can be reached by 5:00, a determination is made asto whether the assertion/promise has been met (decision block 356 ofFIG. 45 and decision block 400 of FIG. 46). For example, in the “I'll behome by 5:00” assertion, the system may determine whether the occupanthas reached home. If the assertion has been met, routine monitoringproceeds (block 358 of FIG. 45 and block 402 of FIG. 46). However, ifthe assertion has not been met, the process 350 returns to decisionblock 354 and process 390 returns to block 392 to determine whether theassertion/promise is capable of being met.

If the assertion/promise cannot be met (e.g., the occupant's housecannot be reached by 5:00), a finding that the assertion cannot be metmay be reported and/or recorded (block 360 of FIG. 45 and block 398 ofFIG. 46). For example, a notification may be provided via an electronicdevice, noting that the assertion/promise can no longer be met.

In some embodiments, intermediate notifications may be reported/recordedwhen an assertion is close to not being able to be met. For example, inthe appointment process 390, a notification could be sent 30 minutes, 15minutes, 5 minutes, etc. prior to the appointment not being able to bekept. Accordingly, the asserter may be prompted to head towards theappointment location prior to not being able to make the appointment.

Household Policy-Based Reporting and Control

Once the monitoring and/or inference data is reported and/or recorded toat the household policy management system, the household policymanagement system may facilitate further household reporting and/orcontrol. FIG. 47 is a flow diagram illustrating a process 420 forproviding reporting and/or control of a smart-device environment, inaccordance with an embodiment. The process 420 begins with receivinghousehold findings (e.g., those findings reported and/or recorded, asdescribed herein) (block 422). Based upon the particular householdpolicies implemented within the household policy management system, thehousehold policy management system may provide reporting of the findings(block 424) and/or provide control based upon the findings (block 426).

Turning first to reporting, notification of findings may be provided asa report via a multitude of notification mechanisms. FIG. 47 provides anexample of several notification mechanisms. For example, notification ofcertain findings may be provided via smart lighting systems 428. Forexample, a household policy may be set to change a smart lighting systemcolor to red and flash the lights whenever findings warranting a warningare received.

Additionally, reporting may be provided via other smart devices. Forexample reporting may be provided via a tablet computing device 430, viaaudio speakers (e.g., in a smart smoke detector 432), via a displaypanel (e.g., of a smart thermostat 434), and/or personal smart devices,such as an activity monitor wristband 436. The reporting can be providedvia social networking sites (e.g., Google+ 437) or other websites. Insome embodiments reporting can be provided to applications. For example,the reporting can be provided to Nest software applications and/orhardware for Nest Leaf 438 feedback (e.g., energy savings feedbackindication from Nest applications and/or hardware).

Turning now to control, the received household findings may be used tocontrol any number of smart devices. For example, the household findingsmight be used to control lock 440 (e.g., door lock) functionality. Forexample, if the findings indicate that children are home alone, thelocks 440 may be actuated to a locked position, such that the house issecured from unexpected guests (assuming a household policy isimplemented for such functionality).

In some embodiments, the received findings may control access to content442 (e.g., Internet-based content, television content, allowed DVDs,etc.). For example, if the findings indicate that chores have not beencompleted and/or an excessive amount of content has been consumed, thehousehold policy manager may restrict access to further content 442(assuming a household policy is implemented for such functionality). Insome embodiments, access to content playback devices (e.g., a television444 and/or computing device 446) may be restricted and/or allowed basedupon findings. For example, when the findings indicate that chores havebeen completed, access to the devices 444 and/or 446 and/or content 442may be granted.

In some embodiments, lighting 448 or other household fixtures may becontrolled based upon the received household findings. For example,lighting 448 may be turned off when the household policy managerobserves that there is no occupancy in a particular room (assuming ahousehold policy is implemented for such functionality). Additionally,controllable household appliances, such as a washer and/or dryer 450 maybe controlled based upon the received findings. For example, if thefindings suggest that a television or radio is being used near thewasher and/or dryer 450, the washer and/or dryer execution cycles may bedelayed, enabling the use of the television and/or radio without noisefrom the washer and/or dryer 450.

The reporting/notification mechanisms and controlled devices mentionedabove are provided merely as examples of how notifications may beprovided and how devices may be controlled. The provided examples arenot intended to limit the scope of notification mechanisms or controlleddevices within the household.

Turning now to particular reporting and/or control policies, FIG. 48 isa flow diagram illustrating a process 470 for situation-based reportingand/or control, in accordance with an embodiment. The process 470 beginsby receiving a situation-based policy (block 472), such as third-partysupervision of children as will be discussed with regard to FIG. 49and/or grounding as will be discussed with regard to FIG. 50).

Once the situation-based household policy is received, the householdpolicy manager begins monitoring for the occurrence of the situationassociated with the policy (e.g., third-party supervision or grounding)(block 474). When the situation is detected, the situation-basedhousehold policy is implemented (e.g., the reporting and/or controlbased upon the policy is implemented) (block 476). The situation-basedhousehold policy is implemented until the situation is no longerdetected (e.g., no longer supervised by the third party). Once thesituation is no longer detected, normal household policy implementationresumes (block 478).

FIG. 49 is a flow diagram illustrating a process 490 for controlling asmart-device environment under third party (e.g., non-parentalsupervision), in accordance with an embodiment. In the embodiment ofFIG. 49, a non-parental supervision policy 492 dictates particularactions to be taken upon detection of non-parental supervision. Forexample, the policy 492 dictates that upon such an occurrence,sub-policy sensitivity should be increased (block 494). For example, ifthere is a policy that, under normal supervision, a notification isprovided to parents when 5 uses of foul language occur, the sensitivitymay be increased, such that parents are notified when 2 uses of foullanguage occur under non-parental supervision.

Additionally, the policy 492 dictates that increased monitoring shouldbe implemented when there is supervision by a third party (block 496).For example, increased sensor usage may be warranted, enabling increasedmonitoring with less trusted supervision. The increased sensorutilization may help protect the children and monitor the activities ofthe supervisors as well.

The policy 492 also dictates that device usage and/or device contentshould be limited during third party supervision (block 498). Forexample, while parents may be able to comfort children when exposed toscary content, third party supervisors may not be so well equipped.Accordingly, the policy 492 may dictate limiting scary content (or anyother content) when parents are not supervising the children. Further,the parents may wish for the children to engage with the third partysupervisors rather than their electronic devices. Accordingly, thepolicy 492 may limit electronic device usage when a third partysupervisor is present.

The policy 492 also dictates that specialized notifications should besent when a third party supervises the children (block 500). Forexample, a parent may wish to be notified of the occurrence of certainactivities when a third party is supervising the children, but not whenthe parents are supervising the children. For example, a notificationthat the children are outside may be useful when a third party issupervising the children, but may be unnecessary when parents aresupervising the children. Accordingly, specialized notification (block500) may be implemented upon the occurrence of a situation defined in apolicy.

Turning now to a grounding policy, FIG. 50 is a flow diagramillustrating a process 520 for grounding control of a smart-deviceenvironment, in accordance with an embodiment. In the process 520, agrounding policy 522 is received by the household policy managementsystem. The grounding policy 522 may be activated based upon anindication provided by a household occupant with sufficient privilegesto ground other occupants of the household. The indication may beprovided to the household policy manager, for example, via use of agraphical user interface, audio commands, or both.

Once implemented, the policy 522 may control one or more characteristicsof the household. Control of household characteristics may be controlledin many manners. For example, control may be based upon particularrooms/zones of the house, may be based upon associations with particularoccupants, etc. In the current embodiment, the grounding policy 522dictates that, when active, electronic device usage and/or availableelectronic device content is to be limited in rooms where the groundedoccupant is located (block 524). In some embodiments, these rooms may bedetermined based upon discernment of a particular room/zone the groundedoccupant is occupying. In other embodiments, these rooms may bedetermined based upon rooms that are associated with the groundedoccupant (e.g., a game room and/or the occupant's bedroom).

The grounding policy 522 may also dictate that monitoring of thegrounded occupants location should ensue and notification provided whenthe grounded occupant leaves a particular grounding location. Forexample, the grounded occupant may be commanded to spend their groundingtime in their bedroom alone. The grounding policy 522 may monitor thelocation of the grounded occupant and notify the supervisor when thegrounded occupant's location deviates from the grounded occupant'sbedroom.

The grounding policy 522 may also dictate that a “lights out” modeshould be activated at a particular time (block 528). For example, the“lights out” mode may include: controlling lighting to power off and/orcontrolling electronic devices to power off. This “lights out” mode maybe activated in a particular room/zone the grounded occupant isoccupying and/or a particular room/zone associated with the groundedoccupant (e.g., the grounded occupant's bedroom).

Household Behavioral Chances

FIG. 51 is a flow diagram illustrating a process 540 for reward-basedcontrol of a smart-device environment, in accordance with an embodiment.The process 540 begins by defining a goal (block 542). For example, thesystem may, based upon one or more demographics and/or patterns of ahousehold, provide a suggestion regarding typically desired “areas ofimprovement” available for the household. Based upon these suggestions,the household may select one or more future goals. For example, basedupon observed household patterns, the system may notice that members ofthe household spend less time eating together than other family havingcommon demographics. Based upon this observation, the system may suggestthat the household strive to attain this goal. If the household agrees,the goal is set as defined goal. Other examples of goals might includedecreased energy usage, modifications to allotments of electronic device“screen-time,” reduction of foul language and/or “raised voice”conversations, etc.

The system will then monitor progression towards the goal (block 544).As will be discussed in more detail below, with regard to FIGS. 54-58,the progression toward the goal may be gradual, thus, graduatedperformance toward the end goal may also be monitored. Based upon themonitoring activities, a determination is made as to whether the goalhas been met, partially met, or has not been met (decision block 546).

If no progress is made, a reminder of the active goal may be provided tothe household (block 547). For example, if the household indicated agoal to spend more time doing activities together and the systemmonitoring indicates that the household is spending less time togetheror marginally more time together, the system may provide a reminder ofthe household goal to one or more members of the household (e.g., via anaudible and/or visual alert in the household, via a text messageprovided to the user's smartphone, etc.).

If sufficient progress toward the goal is attained, a progress rewardmay be provided to one or more members of the household (block 548). Forexample, in the togetherness goal mentioned above, if the family spends20 additional minutes together in a week, when the goal is to spend anadditional hour together a week, a progress reward may be provided tothe family. Further, if a portion of the family spends the additionalhour together, but a portion does not, the portion of the membersattaining the goal may receive a progress reward, while the portion ofthe members not attaining the goal does not.

If the goal is attained, a reward for attaining the goal may be providedto one or more of the household members (block 550). For example, if allof the household members spend an additional hour doing activitiestogether, the goal may be attained, thus resulting in the presentationof the reward for attaining the goal.

The progress goal may be as simple as an encouraging text, audio, and/orvideo message, or may be more elaborate. For example, in someembodiments, increase allotments of portion restricted activities may beprovided. For example, an increase in electronic device “screen-time”may be provided upon attaining a goal. In one embodiment, householdbragging rights may be a reward, by providing a neighborhood message toother participating households, stating that the household isprogressing towards and/or has attained the particular goal.

FIG. 52 is a schematic drawing illustrating a system 570 providing anallotment-based reward for attaining a goal, in accordance with anembodiment. In the embodiment of FIG. 52, a household administrator hasdefined a goal that Benjamin should spend more time outside and lesstime on electronic devices.

Accordingly, based upon the defined goal, the system may monitorBenjamin's actions, discerning how Benjamin is allotting his time. Forexample, cameras in the household may determine whether Benjamin isinside or outside the house, whether Benjamin is on an electronicdevice, etc. Further, electronic device usage and/or Benjamin's locationmay be directly attainable using functions of the electronic device 572(e.g., Benjamin's smartphone).

Using this data, the system 570 may determine whether Benjamin isspending more time outside. If the system determines that Benjamin hasnot progress towards spending more time outside, the system may providea reminder, as illustrated in the electronic device 572 at time 574. Insome embodiments, the reminder (or other mechanism in the system) mayencourage progress to the goal by imposing restrictions and/or otherpunishments for not progressing toward the goal. For example, becauseBenjamin has not progressed toward the goal at time 574, the system mayimpose a “screen-time” lockout feature, thus encouraging Benjamin toprogress toward the goal.

Once sufficient progression is made (e.g., Benjamin is outside for 15minutes), a progression goal, such a “screen-time” lockout countdownclock may be presented to Benjamin. Further, once Benjamin's goal isattained (e.g., Benjamin spends 30 minutes outside as indicated at time576 by block 578), the “screen-time” lockout feature of the electronicdevice 572 may be disabled, as shown at time 580. Thus, because Benjaminhas attained the goal, he is allotted additional “screen-time.”

In some embodiments, when goals are allotment-based, some flexibilitymay be implemented by the system. For example, FIG. 53 is a flow diagramillustrating a process 600 for the allotment-based control of thesmart-device environment where carryover allotments may be allowed, inaccordance with an embodiment. First, the system may determine that aportion of an allotment (e.g., an allotment of “screen-time” of anelectronic device) has been budgeted, but not used (block 602). Forexample, if a household member is allotted two hours of “screen-time”total in a day, but only uses one hour, an additional one hour of“screen-time” has been budgeted, but not used.

The system may then determine if the policies implementing the budgetedallotments allow for carryover (decision block 604). For example, insome instances, a policy may be implemented in a more flexible mannerthat allows un-used allotments for one time period to carryover to beapplied to allotments of the next time period. Alternatively, in someembodiments, carryover is not allowed, ensuring that the allotment isthe maximum allotment for the particular time period.

If carryover is not allowed, the standard allotment is maintained (block606). However, if carryover is allowed, the un-used allotment is addedto the budgeted allotment for the next time period (block 608).Accordingly, in the above examiner where the household member does notuse one hour of a budgeted two-hour allotment, the allotment for thenext day will be three hours, because the un-used one hour is added tothe two-hour allotment. This may reduce the rigidity of the policyimplementation, thus, prolonging the effectiveness of the implementedpolicies.

As mentioned above, in some embodiments, gradual progression toward agoal may be an effective way of reaching the goal. For example, bymaking small, incremental progressions toward a goal that are virtuallyimperceptible, the household may become gradually accustomed to theactions of the goal. FIG. 54 is a flow diagram illustrating a process620 for progressive control in a smart-device environment, in accordancewith an embodiment. The process 620 begins by determining the goal to beprogressively implemented (block 622). For example, some goals may beeasier to implement in a progressive manner than others. Goals relatedto changing highly routine activities that occur on a consistent basismay be good candidates for progressive implementation. Further, goalsrelated to household modifications of a significant magnitude may alsobe good candidates for progressive implementation.

After the goals for progressive implementation are determined, thesystem may determine acceptable progression thresholds for theparticular goal (block 624). For example, when there is a pre-definedtime to reach the goal, the progression thresholds may exponentiallygraduate, such that the progression requirements are very subtle atfirst and then more aggressive as the household becomes accustomed toprogressing toward the goal. Alternatively, the progression thresholdsmay be evenly distributed amongst time periods between implementation ofthe goal and the pre-defined time to reach the goal, such that an equalamount of progression is to be made during each time period.

In embodiments where no pre-defined time to reach the goal is specified,the system may increase the progression requirement very gradually,providing virtual imperceptibility that the progression is occurring.This may result in additional success in achieving household goals, byallowing a household to adapt very gradually to change.

Based upon the determined acceptable progression for the particulargoal, the system may define progression threshold triggers that are usedto report progression and/or control the system (block 626). Forexample, using an example where a household wishes to decrease Internetusage, an allotment may be reduced by one minute each day, thus beingvirtually imperceptible to the household. The triggers for restrictingInternet access may be moved by one minute each day until a desiredamount of Internet usage is attained. Once the triggers are determined,goal-based policies are implemented based upon the triggers (block 628).Accordingly, the reporting and/or control may become progressivelysensitive, as the household progresses toward the goal.

FIG. 55 is a data chart 650 illustrating an example of control viaprogressively increasing control thresholds, in accordance with anembodiment. Line 652 represents threshold trigger points (e.g., they-axis 654) as a function of time (e.g., the x-axis 656). Asillustrated, the line 652 increases gradually until the goal 658 isreached. Thus, reporting and/or control conducted by the system maygradually enforce progression to the goal 658.

This may be seen in the embodiment of FIG. 56. FIG. 56 is a schematicdrawing illustrating a system 670 that provides rewards that becomeprogressively difficult to receive as household members progress towardsa goal, in accordance with an embodiment.

The goal 658 in FIG. 56 is that a particular household member be on timeto a particular destination (e.g., work and/or home) 5 times a week. Toimplement policies for attaining the goal 658 in a progressive fashion,the system 670 may, at time 656A, set a threshold 656A for receiving areward (e.g., praise message 672A) to 1 (e.g. indicating that the praisemessage 672A will be received upon the household member being on timeonce). As the household member progresses 674 toward the goal (e.g., ison time once 676), the threshold may be increased. For example, at time656B, threshold 654B requires being on time twice 678 to receive thepraise message 672B. Thus, the household member may be subtly andgradually encouraged to continue to progress toward the goal 658.

FIG. 57 is a data chart 680 illustrating control via a progressivelydecreasing control threshold to reach a goal 658, in accordance with anembodiment. Line 652 represents threshold trigger points (e.g., they-axis 654) as a function of time (e.g., the x-axis 656). Asillustrated, the line 652 increases gradually until the goal 658 isreached. Thus, reporting and/or control conducted by the system maygradually enforce progression to the goal 658. In contrast to data chart650 of FIG. 55, the data chart 680 has progressively decreasing triggerpoints as a function of time.

This may be seen in the embodiment of FIG. 58. FIG. 58 is a schematicdrawing illustrating a system 710 that progressively removes Internetallotment for household members over time, in accordance with anembodiment.

The goal 658 in FIG. 56 is that a household reduce Internet usage to amaximum of three hours per day. To implement policies for attaining thegoal 658 in a progressive fashion, the system 710 may, at time 656A, seta threshold 656A for controlling a household attribute (e.g., Internetaccess) to 7 (e.g. indicating that Internet access will be revoked after7 hours of usage is reached). As the household progresses 674 toward thegoal (e.g., begins reducing Internet usage), the threshold may bedecreased. For example, at time 656B, threshold 654B is set to 3 hours,indicating that the maximum allotment of Internet usage is now threehours. Any granularity of threshold modifications may be used to attainthe goal 658. For example, thresholds may be modified, for example,every year, month, week, day, hour, etc. Further, the degree ofthreshold modification may vary, depending on any number of factors. Forexample, for goals that will be hard to achieve, each thresholdmodification may be of a very small degree. Thus, the household may besubtly and gradually encouraged to continue to progress toward the goal658.

As may be appreciated, gradual progression toward a goal may effectivelyhelp households and/or household members attain group and/or individualgoals. The systems provided herein encourage such goal attainment byprogressively challenging households and/or household members tocontinue to progress toward completion of a goal. The systems providedherein may apply to any number of goals. One such type of goals areallotment modification goals, such as: decreasing household energyusage, reducing caloric intake of a household member and/or the entirehousehold, reducing Internet time, increasing time outdoors, reducing“raise voice” conversations, decreasing the use of foul language, etc.

The above-described embodiments are directed to achieving what can bethought of as a conscious home, a conscientious home, a thoughtful home,or more generally a smarter home than home automation systems that aresimply based on if-X-occurs-then-do-Y (IXOTDY) programming by the user.The above-described embodiments are directed to providing moreintelligent governance of the home experience by taking into account thefact that information gathered by smart-home sensors is often incompleteand imperfect, that user behaviors and desires for automated assistancefrom the home are often not subject to fixed explicit formulation, andthat the home environment is ever-changing in terms of human goals,human behavior, and the ever-increasing variety and availability of homesensors and controller devices. Thus, in contrast to implementation bydirect IXOTDY rules, the household policy implementations according toof one or more of the embodiments take into account that the triggeringof certain actions or events should be based on a more thoughtfulapproach that takes into account the described confidences, inferences,trends, and other factors to provide a more comprehensive, stable,sustainable, context-aware home automation experience. As one of manyadvantages, it is believed that household policy implementationsaccording to one or more embodiments reduces or obviates the need forpersistent user attention to the building and maintenance of complexIXOTDY rule sets, an all-too-common and time-consuming process which isbelieved by the present inventors to be antithetical to the true purposeof the conscious home, which is to make people's lives easier, better,and more sustainable. In view of the present disclosure, one skilled inthe art would be readily able to implement the described systems andmethods using one or more known technologies, platforms, models, and/ormathematical strategies including, but not limited to, artificial neuralnetworks, Bayesian networks, genetic programming, inductive logicprogramming, support vector machines, decision tree learning, clusteringanalysis, dynamic programming, stochastic optimization, linearregression, quadratic regression, binomial regression, logisticregression, simulated annealing, and other learning, forecasting, andoptimization techniques.

The specific embodiments described above have been shown by way ofexample, and it should be understood that these embodiments may besusceptible to various modifications and alternative forms. It should befurther understood that the claims are not intended to be limited to theparticular forms disclosed, but rather to cover all modifications,equivalents, and alternatives falling within the spirit and scope ofthis disclosure.

What is claimed is:
 1. A method for implementing a household policywithin a household environment, comprising: receiving, at a processor,the household policy, wherein the household policy comprises statisticsassociated with the household environment; interpreting the householdpolicy to extract one or more conditional events associated with thehousehold policy; monitoring, via at least one sensing smart device inthe household environment, for satisfaction of the one or moreconditional events, wherein the one or more conditional events is notdirectly observable via the at least one sensing smart device, andwherein the monitoring comprises: receiving contextual data from each ofthe at least one sensing smart device; and inferring, based on thereceived contextual data and the statistics associated with thehousehold environment, whether the one or more conditional events, whichis not directly observable via the at least one sensing smart device, issatisfied; and when the one or more conditional events, which is notdirectly observable via the at least one sensing smart device, issatisfied, implement one or more controls on at least one conditionallycontrolled smart device in the household environment, the at least oneconditionally controlled smart device affecting the householdenvironment.
 2. The method of claim 1, wherein the one or more controlscomprise posting information to a digital dashboard, a mobileapplication, transitioning one or more device indicators, or acombination thereof.
 3. The method of claim 1, wherein the householdpolicy comprises a virtual grounding policy wherein internet usage isrestricted or limited to particular websites in a particular room of thehousehold environment.
 4. The method of claim 1, wherein the householdpolicy relates to a latchkey child situation where a child is in thehousehold environment alone; and wherein the one or more controlscomprise controls restricting: access to particular content outside thepresence of adult supervision, an ability to unlock a door when anunknown person approaches the door, or both.
 5. The method of claim 1,wherein the one or more conditional events comprises meeting a thresholdamount of study time, a threshold amount outside time, or both.
 6. Themethod of claim 5, wherein the one or more controls comprise increasingan allotment of screen time for at least one electronic device.
 7. Themethod of claim 1, wherein the one or more conditional events comprisean individual having unused screen time from a budgeted screen timeallotted to the individual over a first time period; and wherein the oneor more controls comprise allocating the unused screen time to theindividual for use over a second time period.
 8. The method of claim 1,wherein the one or more conditional events comprise an individual havingused all screen time allotted to the individual over a first timeperiod; and wherein the one or more controls comprise restricting accessof the individual to one or more electronic devices during the remainderof the first time period.
 9. The method of claim 1, wherein the one ormore electronic devices comprises a television.
 10. A system,comprising: a processor, configured to: receive a household policy,wherein the household policy comprises statistics associated with ahousehold environment; and interpret the household policy to extract oneor more conditional events associated with the household policy; atleast one sensing smart device configured to monitor for satisfaction ofthe one or more conditional events, wherein the one or more conditionalevents is not directly observable via the at least one sensing smartdevice; and at least one controlled smart device, configured to operatebased upon one or more control commands provided by the processor whenthe one or more conditional events are satisfied or not satisfied, andwherein the processor is configured to: receive contextual data fromeach of the at least one sensing smart device; and infer, based on thereceived contextual data and the statistics associated with thehousehold environment, whether the one or more conditional events, whichis not directly observable via the at least one sensing smart device, issatisfied.
 11. The system of claim 10, wherein: the processor isconfigured to: receive a situational household policy to be implementedonly upon the occurrence of a particular situation; interpret thesituational household policy to extract the particular situation and oneor more situational conditional events associated with the situationalhousehold policy; the sensing smart device is configured to: monitor foroccurrence of the particular situation; and upon occurrence of theparticular situation, monitor for satisfaction of the one or moresituational conditional events; and the processor is configured toprovide one or more situational controls to the at least oneconditionally controlled smart device when the one or more situationalconditional events is satisfied.
 12. The system of claim 11, wherein theparticular situation comprises non-parent supervision of a child. 13.The system of claim 12, wherein the one or more situational conditionalevents comprise the one or more conditional events with an increasedsensitivity.
 14. The system of claim 12, wherein conditionallycontrolled smart device comprise an electronic device; and the one ormore situational controls are configured to limit usage of theelectronic device, limit content that may be consumed on the electronicdevice, or both.
 15. The system of claim 12, wherein the one or moresituational controls are configured to provide a specializednotification regarding the child, the non-parent supervision, or both.16. The system of claim 11, wherein the particular situation comprisesgrounding of a household occupant.
 17. The system of claim 16, whereinthe one or more situational controls are configured to: limit electronicdevice usage, electronic device content, or both in a particular area ofthe household environment associated with the grounded householdoccupant; limit electronic device usage, electronic device content, orboth attributed to the grounded household occupant, or both.
 18. Thesystem of claim 11, wherein the one or more situational controls areconfigured to: reduce lighting in a particular area of the householdenvironment associated with the grounded household occupant at a “lightsout” time indicated in the situational household policy.